Re: [ClusterLabs] Can't See Why This Cluster Failed Over

2017-04-10 Thread Kristoffer Grönlund 
Eric Robinson  writes:

>> crm configure show xml c_clust19
>
> Here is what I am entering using crmsh (version 2.0-1):
>
>
> colocation c_clust19 inf: [ p_mysql_057 p_mysql_092 p_mysql_187 ] 
> p_vip_clust19 p_fs_clust19 p_lv_on_drbd0 ms_drbd0:Master
> order o_clust19 inf: ms_drbd0:promote p_lv_on_drbd0 p_fs_clust19 
> p_vip_clust19 [ p_mysql_057 p_mysql_092 p_mysql_187 ]
>
>
> After I save it, I get no errors, but it converts it to this...
>
>
> colocation c_clust19 inf: [ p_mysql_057 p_mysql_092 p_mysql_187 ] ( 
> p_vip_clust19:Master p_fs_clust19:Master p_lv_on_drbd0:Master ) ( 
> ms_drbd0:Master )
> order o_clust19 inf: ms_drbd0:promote ( p_lv_on_drbd0:start 
> p_fs_clust19:start p_vip_clust19:start ) [ p_mysql_057 p_mysql_092 
> p_mysql_187 ]
>
> This looks incorrect to me.
>
> Here is the xml that it generates.
>
> 
>   
> 
>   
>   
>   
>   
>   
>   
>   
>   
>   
>   
>   
>   
>   
> 
> 
>   
>   
>   
> 
> 
>   
> 
>   
> 
>
> The resources in set c_clust19-1 should start sequentially, starting with 
> p_lv_on_drbd0 and ending with p_vip_clust19. I also don't understand why 
> p_lv_on_drbd0 and p_vip_clust19 are getting the Master designation. 

Hi,

Yeah, that does indeed look like a bug.. One thing that is confusing and
may be one reason why things get split in an unexpected way is because
as you can see, the role attribute is applied per resource set, while
it looks like it applies per resource in the crmsh syntax. So the shell
does some complex logic to "split" sets based on role assignment.

Cheers,
Kristoffer

>
> --
> Eric Robinson
>
>

-- 
// Kristoffer Grönlund
// kgronl...@suse.com

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Can't See Why This Cluster Failed Over

2017-04-10 Thread Eric Robinson 
> crm configure show xml c_clust19

Here is what I am entering using crmsh (version 2.0-1):


colocation c_clust19 inf: [ p_mysql_057 p_mysql_092 p_mysql_187 ] p_vip_clust19 
p_fs_clust19 p_lv_on_drbd0 ms_drbd0:Master
order o_clust19 inf: ms_drbd0:promote p_lv_on_drbd0 p_fs_clust19 p_vip_clust19 
[ p_mysql_057 p_mysql_092 p_mysql_187 ]


After I save it, I get no errors, but it converts it to this...


colocation c_clust19 inf: [ p_mysql_057 p_mysql_092 p_mysql_187 ] ( 
p_vip_clust19:Master p_fs_clust19:Master p_lv_on_drbd0:Master ) ( 
ms_drbd0:Master )
order o_clust19 inf: ms_drbd0:promote ( p_lv_on_drbd0:start p_fs_clust19:start 
p_vip_clust19:start ) [ p_mysql_057 p_mysql_092 p_mysql_187 ]

This looks incorrect to me.

Here is the xml that it generates.


  

  
  
  
  
  
  
  
  
  
  
  
  
  


  
  
  


  

  


The resources in set c_clust19-1 should start sequentially, starting with 
p_lv_on_drbd0 and ending with p_vip_clust19. I also don't understand why 
p_lv_on_drbd0 and p_vip_clust19 are getting the Master designation. 

--
Eric Robinson
   

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Can't See Why This Cluster Failed Over

2017-04-09 Thread Kristoffer Grönlund 
Eric Robinson  writes:

> Here's the config. I don't know why the CRM put in the parenthesis where it 
> did. That's not the way I typed it. I usually have all my mysql instances 
> between parenthesis and everything else outside.

[ ...]

> colocation c_clust19 inf: ( p_mysql_057 p_mysql_092 p_mysql_187 p_mysql_213 
> p_mysql_250 p_mysql_289 p_mysql_312 p_vip_clust19 p_mysql_702 p_mysql_743 
> p_mysql_745 p_mysql_746 p_fs_clust19 p_lv_on_drbd0 ) ( ms_drbd0:Master )
> colocation c_clust20 inf: p_vip_clust20 p_fs_clust20 p_lv_on_drbd1 
> ms_drbd1:Master
> order o_clust19 inf: ms_drbd0:promote ( p_lv_on_drbd0:start ) ( p_fs_clust19 
> p_vip_clust19 ) ( p_mysql_057 p_mysql_092 p_mysql_187 p_mysql_213 p_mysql_250 
> p_mysql_289 p_mysql_312 p_mysql_702 p_mysql_743 p_mysql_745 p_mysql_746 )

This might be a bug in crmsh: What was the expression you intended to
write, and which version of crmsh do you have?

You can see the resulting XML that crmsh generates and then re-parses
into the line syntax using

crm configure show xml c_clust19

Cheers,
Kristoffer

-- 
// Kristoffer Grönlund
// kgronl...@suse.com

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Can't See Why This Cluster Failed Over

2017-04-07 Thread Ken Gaillot 
On 04/07/2017 12:58 PM, Eric Robinson wrote:
> Somebody want to look at this log and tell me why the cluster failed over? 
> All we did was add a new resource. We've done it many times before without 
> any problems.
> 
> --
> 
> Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:  
>   Forwarding cib_apply_diff operation for section 'all' to master 
> (origin=local/cibadmin/2)
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
> --- 0.605.2 2
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
> +++ 0.607.0 65654c97e62cd549f22f777a5290fe3a
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
> /cib:  @epoch=607, @num_updates=0
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/configuration/resources:   type="mysql_745"/>
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/configuration/resources:   type="mysql_746"/>
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']:
>   
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']:
>   
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']:
>   
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']:
>   
> Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:  
>   Completed cib_apply_diff operation for section 'all': OK (rc=0, 
> origin=ha14a/cibadmin/2, version=0.607.0)
> Apr 03 08:50:30 [22762] ha14acib: info: write_cib_contents:   
>   Archived previous version as /var/lib/pacemaker/cib/cib-36.raw
> Apr 03 08:50:30 [22762] ha14acib: info: write_cib_contents:   
>   Wrote version 0.607.0 of the CIB to disk (digest: 
> 1afdb9e480f870a095aa9e39719d29c4)
> Apr 03 08:50:30 [22762] ha14acib: info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.DkIgSs 
> (digest: /var/lib/pacemaker/cib/cib.hPwa66)
> Apr 03 08:50:30 [22764] ha14a   lrmd: info: 
> process_lrmd_get_rsc_info:  Resource 'p_mysql_745' not found (17 active 
> resources)
> Apr 03 08:50:30 [22764] ha14a   lrmd: info: 
> process_lrmd_rsc_register:  Added 'p_mysql_745' to the rsc list (18 active 
> resources)
> Apr 03 08:50:30 [22767] ha14a   crmd: info: do_lrm_rsc_op:  
> Performing key=10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 
> op=p_mysql_745_monitor_0
> Apr 03 08:50:30 [22764] ha14a   lrmd: info: 
> process_lrmd_get_rsc_info:  Resource 'p_mysql_746' not found (18 active 
> resources)
> Apr 03 08:50:30 [22764] ha14a   lrmd: info: 
> process_lrmd_rsc_register:  Added 'p_mysql_746' to the rsc list (19 active 
> resources)
> Apr 03 08:50:30 [22767] ha14a   crmd: info: do_lrm_rsc_op:  
> Performing key=11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 
> op=p_mysql_746_monitor_0
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
> --- 0.607.0 2
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
> +++ 0.607.1 (null)
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
> /cib:  @num_updates=1
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
> /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources:  
> 
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++
> 
>  operation="monitor" crm-debug-origin="do_update_resource" 
> crm_feature_set="3.0.9" 
> transition-key="13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" 
> transition-magic="0:7;13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" 
> call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++
>   
> 
> Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:  
>   Completed cib_modify operation for section status: OK (rc=0, 
> origin=ha14b/crmd/7665, version=0.607.1)
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
> --- 0.607.1 2
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
> +++ 0.607.2 (null)
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
> /cib:  @num_updates=2
> Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
>

Re: [ClusterLabs] Can't See Why This Cluster Failed Over

2017-04-07 Thread Eric Robinson 
Somebody want to look at this log and tell me why the cluster failed over? All 
we did was add a new resource. We've done it many times before without any 
problems.

--

Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Forwarding cib_apply_diff operation for section 'all' to master 
(origin=local/cibadmin/2)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
--- 0.605.2 2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
+++ 0.607.0 65654c97e62cd549f22f777a5290fe3a
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
/cib:  @epoch=607, @num_updates=0
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/resources:  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/resources:  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Completed cib_apply_diff operation for section 'all': OK (rc=0, 
origin=ha14a/cibadmin/2, version=0.607.0)
Apr 03 08:50:30 [22762] ha14acib: info: write_cib_contents: 
Archived previous version as /var/lib/pacemaker/cib/cib-36.raw
Apr 03 08:50:30 [22762] ha14acib: info: write_cib_contents: 
Wrote version 0.607.0 of the CIB to disk (digest: 
1afdb9e480f870a095aa9e39719d29c4)
Apr 03 08:50:30 [22762] ha14acib: info: retrieveCib:Reading 
cluster configuration from: /var/lib/pacemaker/cib/cib.DkIgSs (digest: 
/var/lib/pacemaker/cib/cib.hPwa66)
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_get_rsc_info:  
Resource 'p_mysql_745' not found (17 active resources)
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_rsc_register:  
Added 'p_mysql_745' to the rsc list (18 active resources)
Apr 03 08:50:30 [22767] ha14a   crmd: info: do_lrm_rsc_op:  
Performing key=10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 
op=p_mysql_745_monitor_0
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_get_rsc_info:  
Resource 'p_mysql_746' not found (18 active resources)
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_rsc_register:  
Added 'p_mysql_746' to the rsc list (19 active resources)
Apr 03 08:50:30 [22767] ha14a   crmd: info: do_lrm_rsc_op:  
Performing key=11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 
op=p_mysql_746_monitor_0
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
--- 0.607.0 2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
+++ 0.607.1 (null)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
/cib:  @num_updates=1
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources:  

Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++  
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0, 
origin=ha14b/crmd/7665, version=0.607.1)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
--- 0.607.1 2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
+++ 0.607.2 (null)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
/cib:  @num_updates=2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources:  

Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++  
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0, 
origin=ha14b/crmd/7666, version=0.607.2)
Apr 03 08:50:30 [22767] ha14a   crmd:   notice: process_lrm_event:  
Operation p_mysql_745_monitor_0: not running (node=ha14a, call=142, rc=7, 
cib-update=88, confirmed=true)
Apr 03 08:50:30 [22767] ha14a   crmd:   notice: process_lrm_event:  
ha14a-p_mysql_745_monitor_0:142 [ not started\n ]
Apr 03 08:50:30 [22762]

[ClusterLabs] Can't See Why This Cluster Failed Over

2017-04-07 Thread Eric Robinson 
Somebody want to look at this log and tell me why the cluster failed over? All 
we did was add a new resource. We've done it many times before without any 
problems.

--

Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Forwarding cib_apply_diff operation for section 'all' to master 
(origin=local/cibadmin/2)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
--- 0.605.2 2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
+++ 0.607.0 65654c97e62cd549f22f777a5290fe3a
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
/cib:  @epoch=607, @num_updates=0
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/resources:  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/resources:  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']:
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Completed cib_apply_diff operation for section 'all': OK (rc=0, 
origin=ha14a/cibadmin/2, version=0.607.0)
Apr 03 08:50:30 [22762] ha14acib: info: write_cib_contents: 
Archived previous version as /var/lib/pacemaker/cib/cib-36.raw
Apr 03 08:50:30 [22762] ha14acib: info: write_cib_contents: 
Wrote version 0.607.0 of the CIB to disk (digest: 
1afdb9e480f870a095aa9e39719d29c4)
Apr 03 08:50:30 [22762] ha14acib: info: retrieveCib:Reading 
cluster configuration from: /var/lib/pacemaker/cib/cib.DkIgSs (digest: 
/var/lib/pacemaker/cib/cib.hPwa66)
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_get_rsc_info:  
Resource 'p_mysql_745' not found (17 active resources)
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_rsc_register:  
Added 'p_mysql_745' to the rsc list (18 active resources)
Apr 03 08:50:30 [22767] ha14a   crmd: info: do_lrm_rsc_op:  
Performing key=10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 
op=p_mysql_745_monitor_0
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_get_rsc_info:  
Resource 'p_mysql_746' not found (18 active resources)
Apr 03 08:50:30 [22764] ha14a   lrmd: info: process_lrmd_rsc_register:  
Added 'p_mysql_746' to the rsc list (19 active resources)
Apr 03 08:50:30 [22767] ha14a   crmd: info: do_lrm_rsc_op:  
Performing key=11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 
op=p_mysql_746_monitor_0
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
--- 0.607.0 2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
+++ 0.607.1 (null)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
/cib:  @num_updates=1
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources:  

Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++  
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0, 
origin=ha14b/crmd/7665, version=0.607.1)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
--- 0.607.1 2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: Diff: 
+++ 0.607.2 (null)
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: +  
/cib:  @num_updates=2
Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++ 
/cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources:  

Apr 03 08:50:30 [22762] ha14acib: info: cib_perform_op: ++  
  
Apr 03 08:50:30 [22762] ha14acib: info: cib_process_request:
Completed cib_modify operation for section status: OK (rc=0, 
origin=ha14b/crmd/7666, version=0.607.2)
Apr 03 08:50:30 [22767] ha14a   crmd:   notice: process_lrm_event:  
Operation p_mysql_745_monitor_0: not running (node=ha14a, call=142, rc=7, 
cib-update=88, confirmed=true)
Apr 03 08:50:30 [22767] ha14a   crmd:   notice: process_lrm_event:  
ha14a-p_mysql_745_monitor_0:142 [ not started\n ]
Apr 03 08:50:30 [22762]