Re: [Linux-HA] disallowing concurrent configuration (CIB modifications)
On Wed, Jul 03, 2013 at 12:42:55AM +0300, Vladislav Bogdanov wrote: 02.07.2013 20:05, Dejan Muhamedagic wrote: On Tue, Jul 02, 2013 at 11:05:01AM +0300, Vladislav Bogdanov wrote: 28.06.2013 17:47, Dejan Muhamedagic wrote: ... If you want to test here's a new patch. It does work with unrelated changes happening in the meantime. I didn't test yet really concurrent updates. One thing I see immediately, is that node utilization attributes are deleted after I do 'load update' with empty node utilization sections. That is probably not specific to this patch. Right. I have that attributes dynamic, set from a RA (as node configuration may vary, I prefer to detect how much CPU and RAM I have and set utilization accordingly rather then put every hardware change into CIB). On the one hand, I would agree that crmsh does what is intended - if no utilization attributes is set in a config update, then they shoud be removed. Well, thinking more about it, the attributes should be merged. The only trouble is that that would then change the command semantically. Not sure that is expected by most people. How you then delete attributes? Tough call :) Ideas welcome. If you really think about implementing that merging, I would introduce a crmsh config option for that. F.e. node_attr_policy (replace|merge). I'm not very keen on configuration options having effect in a process like this one. It's awkward and easy to forget and then not obvious how to read the current setting. It's not only for the nodes. Attributes of resources should be merged as well. Perhaps to introduce another load method, say merge, which would merge attributes of elements instead of replacing them. Though the use would then get more complex (which seems to be justified here). Cheers, Dejan And default value should be the current one. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Backing out of HA
Hi, On Tue, Jul 02, 2013 at 10:36:37PM +0200, Arnold Krille wrote: Just a small comment, maybe of benefit to others... On Mon, 01 Jul 2013 16:31:13 -0400 William Seligman selig...@nevis.columbia.edu wrote: Poisoned resource This is the one you can directly attribute to my stupidity. I add a new resource to the pacemaker configuration. Even though the pacemaker configuration is syntactically correct, and even though I think I've tested it, in fact the resource cannot run on either node. The most recent example: I created a new virtual domain and tested it. It worked fine. I created the ocf:heartbeat:VirtualDomain resource, verified that crm could parse it, and activated the configuration. However, I had not actually created the domain for the virtual machine; I had typed virsh create ... but not virsh define So I had a resource that could not run. What I'd want to happen is for the poisoned resource to fail, I see lots of error messages, but the remaining resources would continue to run. What actually happens is that resource tries to run on both nodes alternately an infinite number of times (1 times or whatever the value is). Then one of the nodes stoniths the other. The poisoned resource still won't run on the remaining node, so that node tries restarting all the other resources in the pacemaker configuration. That still won't work. This is the reason why I _always_ create new resources with 'is-managed=false' and see what happens. pacemaker then runs a monitoring action without doing anything about the results. Very nice to see if the resource is workable for pacemaker without killing the cluster. If all works (and the normal working day is over) I activate all the resources that are not yet managed... New resources can also be tested on all nodes by crm configure rsctest. The test is completely driven by crmsh and won't disturb the cluster in any way. Cheers, Dejan Have fun, Arnold ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Antw: Re: Backing out of HA
Dejan Muhamedagic deja...@fastmail.fm schrieb am 03.07.2013 um 10:29 in Nachricht 20130703082919.GB4831@walrus.homenet: [...] New resources can also be tested on all nodes by crm configure rsctest. The test is completely driven by crmsh and won't disturb the cluster in any way. Can you sketch the workflow? How are the tests done? Starting or stopping resources can disturb the cluster, BTW. Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] disallowing concurrent configuration (CIB modifications)
On 2013-07-03T00:20:19, Vladislav Bogdanov bub...@hoster-ok.com wrote: I do not edit them. I my setup I generate full crm config with template-based framework. And then you do a load/replace? Tough; yes, that'll clearly overwrite what is already there and added by scripts that more dynamically modify the CIB. Since we don't know your complete merging rules, it's probably easier if your template engine gains hooks to first read the CIB for setting those utilization values. That is very convenient way to f.e stop dozen of resources in one shot for some maintenance. I have special RA which creates ticket on a cluster start and deletes it on a cluster stop. And many resources may depend on that ticket. If you request resource handled by that RA to stop, ticket is revoked and all dependent resources stop. I wouldn't write that RA if I have cluster-wide attributes (which perform like node attributes but for a whole cluster). Right. But. Tickets *are* cluster wide attributes that are meant to control the target-role of many resources depending on them. So you're getting exactly what you need, no? What is missing? Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] PCS and ping resources?
Le 02/07/2013 00:53, Chris Feist a écrit : pcs resource create myping ocf:pacemaker:ping host_list=www.microsoft.com timeout=5 op monitor interval=10 You want to put the host_list and timeout before 'op' because they're options for the ocf:pacemaker:ping resource agent. And maybe ping a more stable host ;-) /troll -- Cheers, Florian Crouzat ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] disallowing concurrent configuration (CIB modifications)
On 2013-07-03T10:26:09, Dejan Muhamedagic deja...@fastmail.fm wrote: Not sure that is expected by most people. How you then delete attributes? Tough call :) Ideas welcome. Set them to an empty string, or a magic #undef value. It's not only for the nodes. Attributes of resources should be merged as well. Perhaps to introduce another load method, say merge, which would merge attributes of elements instead of replacing them. Though the use would then get more complex (which seems to be justified here). Well, that leaves open the question of how higher-level objects (primitives, clones, groups, constraints ...) would be affected/deleted. I'm not sure the complexity is really worth it. Merge rules get *really* complex, quickly. And eventually, one ends with the need to annotate the input with how one wants a merge to be resolved (such as #undef values). Then, one might go the easier way of having the caller tell us what they want changed explicitly, instead of having to figure it out ourselves. The whole I'll magically replace the whole configuration and crmsh will figure it out premise seems broken. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] PCS and ping resources?
Am 03.07.2013 12:03, schrieb Florian Crouzat: Le 02/07/2013 00:53, Chris Feist a écrit : pcs resource create myping ocf:pacemaker:ping host_list=www.microsoft.com timeout=5 op monitor interval=10 You want to put the host_list and timeout before 'op' because they're options for the ocf:pacemaker:ping resource agent. And maybe ping a more stable host ;-) That was a fake; I ping the nexthop gateways Thanx to all for the help, looks as I was a bit dumb. Regards JC ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] disallowing concurrent configuration (CIB modifications)
Hi Lars, On Wed, Jul 03, 2013 at 12:05:17PM +0200, Lars Marowsky-Bree wrote: On 2013-07-03T10:26:09, Dejan Muhamedagic deja...@fastmail.fm wrote: Not sure that is expected by most people. How you then delete attributes? Tough call :) Ideas welcome. Set them to an empty string, or a magic #undef value. It's not only for the nodes. Attributes of resources should be merged as well. Perhaps to introduce another load method, say merge, which would merge attributes of elements instead of replacing them. Though the use would then get more complex (which seems to be justified here). Well, that leaves open the question of how higher-level objects (primitives, clones, groups, constraints ...) would be affected/deleted. I'm not sure the complexity is really worth it. Merge rules get *really* complex, quickly. And eventually, one ends with the need to annotate the input with how one wants a merge to be resolved (such as #undef values). Perhaps I misunderstood the original intention, but the idea was more simple: primitive r1 params p1=v1 p2=v2 meta m1=mv1 primitive r1 params p1=nv1 p3=v3 - merge --- primitive r1 params p1=nv1 p2=v2 p3=v3 meta m1=mv1 If the attribute already exists, then it is overwritten. The existing attributes are otherwise left intact. New attributes are added. Thanks, Dejan Then, one might go the easier way of having the caller tell us what they want changed explicitly, instead of having to figure it out ourselves. The whole I'll magically replace the whole configuration and crmsh will figure it out premise seems broken. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Antw: Re: Backing out of HA
Hi Ulrich, On Wed, Jul 03, 2013 at 11:50:59AM +0200, Ulrich Windl wrote: Dejan Muhamedagic deja...@fastmail.fm schrieb am 03.07.2013 um 10:29 in Nachricht 20130703082919.GB4831@walrus.homenet: [...] New resources can also be tested on all nodes by crm configure rsctest. The test is completely driven by crmsh and won't disturb the cluster in any way. Can you sketch the workflow? How are the tests done? Starting or stopping resources can disturb the cluster, BTW. The resource shouldn't be committed yet. Or, otherwise, it should be unmanaged. At any rate, not controlled by the cluster. Here the help text: $ crm configure help rsctest Test resources with current resource configuration. If no nodes are specified, tests are run on all known nodes. The order of resources is significant: it is assumed that later resources depend on earlier ones. If a resource is multi-state, it is assumed that the role on which later resources depend is master. Tests are run sequentially to prevent running the same resource on two or more nodes. Tests are carried out only if none of the specified nodes currently run any of the specified resources. However, it won't verify whether resources run on the other nodes. Superuser privileges are obviously required: either run this as root or setup the `sudoers` file appropriately. Note that resource testing may take some time. Usage: ... rsctest rsc_id [rsc_id ...] [node_id ...] ... Examples: ... rsctest my_ip websvc rsctest websvc nodeB ... Thanks, Dejan Regards, Ulrich ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] disallowing concurrent configuration (CIB modifications)
03.07.2013 13:00, Lars Marowsky-Bree wrote: On 2013-07-03T00:20:19, Vladislav Bogdanov bub...@hoster-ok.com wrote: I do not edit them. I my setup I generate full crm config with template-based framework. And then you do a load/replace? Tough; yes, that'll clearly overwrite Actually 'load update'. 'replace' doesn't work when resources are running. what is already there and added by scripts that more dynamically modify the CIB. Since we don't know your complete merging rules, it's probably easier if your template engine gains hooks to first read the CIB for setting those utilization values. Probably. But not template framework itself (it is combination of make ans m4 actually, so it is too stupid too lookup CIB). So I'd nee to move that onto next model level (human or controlling framework, which I'm in process of implementing) - but that is actually what I wanted to happen (it breaks the whole idea). So I'd probably just hack crmsh to not touch node utilization attributes if whole 'utilization' part is missing in the update. If/when pacemaker has support for transient utilization attributes, I will move to that. That is very convenient way to f.e stop dozen of resources in one shot for some maintenance. I have special RA which creates ticket on a cluster start and deletes it on a cluster stop. And many resources may depend on that ticket. If you request resource handled by that RA to stop, ticket is revoked and all dependent resources stop. I wouldn't write that RA if I have cluster-wide attributes (which perform like node attributes but for a whole cluster). Right. But. Tickets *are* cluster wide attributes that are meant to control the target-role of many resources depending on them. So you're getting exactly what you need, no? What is missing? They are volatile. And, I'd prefer cluster attributes to have free-form values. I was already hit by the fact that two-state 'granted/revoked' value is too limited for me. I then expanded logic to also use non-existent' ticket state (it worked for some time), but then support for active/standby came in and I switched to that. Tat all was in lustre-server RA, which needs to control order in which parts of a whole lustre fs are tuned/activated when it moves to another cluster on a ticket revocation. I use additional internally-controlled ticket there. Best, Vladislav ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] disallowing concurrent configuration (CIB modifications)
03.07.2013 15:43, Dejan Muhamedagic wrote: Hi Lars, On Wed, Jul 03, 2013 at 12:05:17PM +0200, Lars Marowsky-Bree wrote: On 2013-07-03T10:26:09, Dejan Muhamedagic deja...@fastmail.fm wrote: Not sure that is expected by most people. How you then delete attributes? Tough call :) Ideas welcome. Set them to an empty string, or a magic #undef value. It's not only for the nodes. Attributes of resources should be merged as well. Perhaps to introduce another load method, say merge, which would merge attributes of elements instead of replacing them. Though the use would then get more complex (which seems to be justified here). Well, that leaves open the question of how higher-level objects (primitives, clones, groups, constraints ...) would be affected/deleted. I'm not sure the complexity is really worth it. Merge rules get *really* complex, quickly. And eventually, one ends with the need to annotate the input with how one wants a merge to be resolved (such as #undef values). Perhaps I misunderstood the original intention, but the idea was more simple: primitive r1 params p1=v1 p2=v2 meta m1=mv1 primitive r1 params p1=nv1 p3=v3 - merge --- primitive r1 params p1=nv1 p2=v2 p3=v3 meta m1=mv1 If the attribute already exists, then it is overwritten. The existing attributes are otherwise left intact. New attributes are added. I'd simplify that logic to sections. * node attributes (except pacemaker internal ones like $id?) * node utilization * primitive params * primitive meta * primitive utilization * clone/ms meta If whole section is missing in the update, then leave it as-is. Otherwise (also if it exists but empty) replace the whole section. The only unclear thing is 'op', but this one can be replaced unconditionally (like in the current logic). Vladislav ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] crm node delete
On Tue, Jul 02, 2013 at 07:53:52AM +0300, Vladislav Bogdanov wrote:
01.07.2013 18:29, Dejan Muhamedagic wrote:
Hi,
On Mon, Jul 01, 2013 at 05:29:31PM +0300, Vladislav Bogdanov wrote:
Hi,
I'm trying to look if it is now safe to delete non-running nodes
(corosync 2.3, pacemaker HEAD, crmsh tip).
# crm node delete v02-d
WARNING: 2: crm_node bad format: 7 v02-c
WARNING: 2: crm_node bad format: 8 v02-d
WARNING: 2: crm_node bad format: 5 v02-a
WARNING: 2: crm_node bad format: 6 v02-b
INFO: 2: node v02-d not found by crm_node
INFO: 2: node v02-d deleted
#
So, I expect that crmsh still doesn't follow latest changes to 'crm_node
-l'. Although node seems to be deleted correctly.
For reference, output of crm_node -l is:
7 v02-c
8 v02-d
5 v02-a
6 v02-b
This time the node state was empty. Or it's missing altogether.
I'm not sure how's that supposed to be interpreted. We test the
output of crm_node -l just to make sure that the node is not
online. Perhaps we need to use some other command.
Likely it shows everything from a corosync nodelist.
After I deleted the node from everywhere except corosync, list is still
the same.
OK. This patch changes the interface to crm_node to use the
list partition option (-p). Could you please test it?
Cheers,
Dejan
___
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
# HG changeset patch
# User Dejan Muhamedagic de...@hello-penguin.com
# Date 1372868970 -7200
# Wed Jul 03 18:29:30 2013 +0200
# Node ID d6f915195baceb5d0e3b76297ad1c8188c767e2f
# Parent 2c70ae4d11d3744bb6917170d0216617ec3a11dd
Medium: node: update interface to crm_node (node delete)
diff -r 2c70ae4d11d3 -r d6f915195bac modules/ui.py.in
--- a/modules/ui.py.in Tue Jul 02 12:44:55 2013 +0200
+++ b/modules/ui.py.in Wed Jul 03 18:29:30 2013 +0200
@@ -1258,50 +1258,30 @@ class NodeMgmt(UserInterface):
if not is_our_node(node):
common_err(node %s not found in the CIB % node)
return False
-rc = True
if cluster_stack() == heartbeat:
-rc = ext_cmd(self.hb_delnode%node) == 0
+cmd = (self.hb_delnode % node)
else:
-node_states = {}
-ec, l = stdout2list(%s -l % self.crm_node)
-for s in l:
-a = s.split()
-if len(a) != 3:
-common_warn(%s bad format: %s % (self.crm_node, s))
-continue
-# fmt: id uname status
-# remove only those in state lost
-if a[1] == node:
-node_ref_du_jour = a[0]
-if is_pcmk_118():
-node_ref_du_jour = a[1]
-node_states[a[2]] = 1
-if a[2] == lost:
-ec = ext_cmd(%s --force -R %s % \
-(self.crm_node, node_ref_du_jour))
-if ec != 0:
-common_warn('%s --force -R %s failed, rc=%d' % \
-(self.crm_node, node_ref_du_jour, ec))
-if not node_states:
-common_info(node %s not found by %s % (node, self.crm_node))
-elif member in node_states:
-common_info(node %s appears to be still active % node)
-common_info(check output of %s -l % self.crm_node)
-rc = False
-elif not lost in node_states:
-common_err(node %s's state not recognized: %s % (node, node_states.keys()))
-common_info(check output of %s -l % self.crm_node)
-rc = False
-if rc:
-if ext_cmd(self.node_delete%node) != 0 or \
-ext_cmd(self.node_delete_status%node) != 0:
-common_err(failed to remove %s from the CIB % node)
-rc = False
-if rc:
-common_info(node %s deleted % node)
-else:
-common_err('failed to delete node %s' % node)
-return rc
+ec, s = get_stdout(%s -p % self.crm_node)
+if not s:
+common_err('%s -p could not list any nodes (rc=%d)' % \
+(self.crm_node, ec))
+return False
+partition_l = s.split()
+if node in partition_l:
+common_err(according to %s, node %s is still active % \
+(self.crm_node, node))
+return False
+cmd = %s --force -R %s % (self.crm_node, node)
+ec = ext_cmd(cmd)
+if ec != 0:
+common_warn('%s failed, rc=%d' % (cmd, ec))
+return False
+if ext_cmd(self.node_delete % node) != 0 or \
+ext_cmd(self.node_delete_status % node) != 0:
+common_err(%s removed from
[Linux-HA] Preventing MS from starting on certain nodes in a 4-node cluster
I'm setting up a 4-node cluster where 2 of the nodes will run an active/active DRBD and the other two nodes will be available in case one of the active nodes fails. I can almost get this working fine. But, what I can't seem to figure out is how to restrict the DRBD Masters to only certain combination of nodes. I want the replication to happen between: pzsmbr01 and pzsmbr51 or pzsmbr02 and pzsmbr51 or pzsmbr01 and pzsmbr52 or pzsmbr02 and pzsmbr52 Because I share the SAN disks between the following node combinations, I do NOT want the DRBD Masters to wind up on: pzsmbr01 and pzsmbr02 or pzsmbr51 and pzsmbr52 If there's not an easy way to restrict the MS resource like that and getting rid of the SAN sharing would make this setup easier, then I'd be happy to. But, I must have the ability to move a group of resources around the cluster freely and I don't think I can do that with DRBD stacking (can I???). Thanks for any help. Leland ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
