Some additional information that Florian on IRC helped me dig out from the DC logs. This happens about 6 seconds before every monitor op is cancelled. It definitely seems like my colocation constraints are wrong because from this log it cannot find any node to run the resource on.
The interesting lines are these. Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: ERROR: color_instance: Pre-allocation failed: got app02 instead of app01 Dec 27 13:57:15 app02 pengine: [1494]: info: native_deallocate: Deallocating resOCFS:1 from app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: All nodes for resource resOCFS:1 are unavailable, unclean or shutting down (app01: 1, -1000000) Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Could not allocate a node for resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: info: native_color: Resource resOCFS:1 cannot run anywhere Here is the whole log. I hope someone can point to an error in my CIB that I posted in the previous email and help me solve this. Dec 27 13:57:15 app02 crmd: [1495]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms) Dec 27 13:57:15 app02 crmd: [1495]: debug: s_crmd_fsa: Processing I_PE_CALC: [ state=S_IDLE cause=C_TIMER_POPPED origin=crm_timer_popped ] Dec 27 13:57:15 app02 crmd: [1495]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Dec 27 13:57:15 app02 crmd: [1495]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Dec 27 13:57:15 app02 crmd: [1495]: debug: do_state_transition: All 2 cluster nodes are eligible to run resources. Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_DC_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_INTEGRATE_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_FINALIZE_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_PE_INVOKE Dec 27 13:57:15 app02 crmd: [1495]: debug: do_pe_invoke: Query 659: Requesting the current CIB: S_POLICY_ENGINE Dec 27 13:57:15 app02 pengine: [1494]: info: unpack_config: Startup probes: enabled Dec 27 13:57:15 app02 pengine: [1494]: debug: unpack_config: STONITH timeout: 60000 Dec 27 13:57:15 app02 pengine: [1494]: debug: unpack_config: STONITH of failed nodes is disabled Dec 27 13:57:15 app02 pengine: [1494]: debug: unpack_config: Stop all active resources: false Dec 27 13:57:15 app02 crmd: [1495]: debug: do_pe_invoke_callback: Invoking the PE: query=659, ref=pe_calc-dc-1356613035-752, seq=200, quorate=1 Dec 27 13:57:15 app02 pengine: [1494]: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default Dec 27 13:57:15 app02 pengine: [1494]: debug: unpack_config: Default stickiness: 0 Dec 27 13:57:15 app02 pengine: [1494]: notice: unpack_config: On loss of CCM Quorum: Ignore Dec 27 13:57:15 app02 pengine: [1494]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Dec 27 13:57:15 app02 pengine: [1494]: info: unpack_domains: Unpacking domains Dec 27 13:57:15 app02 pengine: [1494]: info: determine_online_status: Node app02 is online Dec 27 13:57:15 app02 pengine: [1494]: info: determine_online_status: Node app01 is online Dec 27 13:57:15 app02 pengine: [1494]: debug: unpack_rsc_op: resOCFS:0_last_failure_0 on app02 returned 0 (ok) instead of the expected value: 7 (not running) Dec 27 13:57:15 app02 pengine: [1494]: info: unpack_rsc_op: Operation monitor found resource resOCFS:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: info: find_clone: Internally renamed resO2CB:0 on app01 to resO2CB:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: find_clone: Created orphan for cloneOcfs2Mgmt: resDLM:0 on app01 Dec 27 13:57:15 app02 pengine: [1494]: info: find_clone: Internally renamed resDLM:0 on app01 to resDLM:2 (ORPHAN) Dec 27 13:57:15 app02 pengine: [1494]: info: native_print: VirtualIP0#011(ocf::heartbeat:IPaddr2):#011Started app01 Dec 27 13:57:15 app02 pengine: [1494]: info: clone_print: Master/Slave Set: msDrbdShared0 [resDrbdShared0] Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDrbdShared0:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDrbdShared0:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDrbdShared0:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDrbdShared0:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: info: short_print: Masters: [ app02 app01 ] Dec 27 13:57:15 app02 pengine: [1494]: info: clone_print: Clone Set: cloneOCFS [resOCFS] Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resOCFS:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resOCFS:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resOCFS:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resOCFS:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: info: short_print: Started: [ app02 app01 ] Dec 27 13:57:15 app02 pengine: [1494]: info: clone_print: Clone Set: cloneJboss [resJboss] (unmanaged) Dec 27 13:57:15 app02 pengine: [1494]: info: short_print: Stopped: [ resJboss:0 resJboss:1 ] Dec 27 13:57:15 app02 pengine: [1494]: info: clone_print: Clone Set: cloneOcfs2Mgmt [groupOcfs2Mgmt] Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDLM:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resO2CB:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDLM:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resO2CB:0 active on app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDLM:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resO2CB:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resDLM:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_active: Resource resO2CB:1 active on app01 Dec 27 13:57:15 app02 pengine: [1494]: info: short_print: Started: [ app02 app01 ] Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app01 to VirtualIP0 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app01 to resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_color: Allocated 2 msDrbdShared0 instances of a possible 2 Dec 27 13:57:15 app02 pengine: [1494]: debug: master_color: resDrbdShared0:0 master score: 10000 Dec 27 13:57:15 app02 pengine: [1494]: info: master_color: Promoting resDrbdShared0:0 (Master app02) Dec 27 13:57:15 app02 pengine: [1494]: debug: master_color: resDrbdShared0:1 master score: 10000 Dec 27 13:57:15 app02 pengine: [1494]: info: master_color: Promoting resDrbdShared0:1 (Master app01) Dec 27 13:57:15 app02 pengine: [1494]: info: master_color: msDrbdShared0: Promoted 2 instances of a possible 2 to master Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app01 to resDLM:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app01 to resO2CB:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resDLM:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resO2CB:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_color: Child groupOcfs2Mgmt:2 not allocated - limit reached Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_color: Allocated 2 cloneOcfs2Mgmt instances of a possible 2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:1 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: ERROR: color_instance: Pre-allocation failed: got app02 instead of app01 Dec 27 13:57:15 app02 pengine: [1494]: info: native_deallocate: Deallocating resOCFS:1 from app02 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Assigning app02 to resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_rsc_colocation_rh: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: All nodes for resource resOCFS:1 are unavailable, unclean or shutting down (app01: 1, -1000000) Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Could not allocate a node for resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: info: native_color: Resource resOCFS:1 cannot run anywhere Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_color: Allocated 1 cloneOCFS instances of a possible 2 Dec 27 13:57:15 app02 pengine: [1494]: info: native_color: Unmanaged resource resJboss:0 allocated to 'nowhere': inactive Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Could not allocate a node for resJboss:0 Dec 27 13:57:15 app02 pengine: [1494]: info: native_color: Unmanaged resource resJboss:1 allocated to 'nowhere': inactive Dec 27 13:57:15 app02 pengine: [1494]: debug: native_assign_node: Could not allocate a node for resJboss:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_color: Allocated 0 cloneJboss instances of a possible 2 Dec 27 13:57:15 app02 pengine: [1494]: debug: master_create_actions: Creating actions for msDrbdShared0 Dec 27 13:57:15 app02 pengine: [1494]: debug: RecurringOp: <null>#011 resJboss:0_monitor_120000 (cancelled : start un-runnable) Dec 27 13:57:15 app02 pengine: [1494]: debug: RecurringOp: <null>#011 resJboss:1_monitor_120000 (cancelled : start un-runnable) Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:0_promote_0 -> groupOcfs2Mgmt:0_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:1_promote_0 -> groupOcfs2Mgmt:1_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:1_promote_0 -> groupOcfs2Mgmt:2_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:0_promote_0 -> groupOcfs2Mgmt:0_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:1_promote_0 -> groupOcfs2Mgmt:1_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:1_promote_0 -> groupOcfs2Mgmt:2_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:0_promote_0 -> groupOcfs2Mgmt:0_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:1_promote_0 -> groupOcfs2Mgmt:1_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resDrbdShared0:1_promote_0 -> groupOcfs2Mgmt:2_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:0_running_0 -> resOCFS:0_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:1_running_0 -> resOCFS:1_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:0_running_0 -> resOCFS:0_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:1_running_0 -> resOCFS:1_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:0_running_0 -> resOCFS:0_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:1_running_0 -> resOCFS:1_start_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resOCFS:0_stop_0 -> groupOcfs2Mgmt:0_stop_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resOCFS:1_stop_0 -> groupOcfs2Mgmt:1_stop_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resOCFS:1_stop_0 -> groupOcfs2Mgmt:2_stop_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resOCFS:0_stop_0 -> groupOcfs2Mgmt:0_stop_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resOCFS:1_stop_0 -> groupOcfs2Mgmt:1_stop_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for resOCFS:1_stop_0 -> groupOcfs2Mgmt:2_stop_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resOCFS:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:0_stopped_0 -> resDrbdShared0:0_demote_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:1_stopped_0 -> resDrbdShared0:1_demote_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:0_stopped_0 -> resDrbdShared0:0_demote_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Created constraint for groupOcfs2Mgmt:1_stopped_0 -> resDrbdShared0:1_demote_0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:0 with groupOcfs2Mgmt:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing resDrbdShared0:1 with groupOcfs2Mgmt:2 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resOCFS:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resOCFS:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:0 with resDrbdShared0:0 Dec 27 13:57:15 app02 pengine: [1494]: debug: clone_update_actions_interleave: Pairing groupOcfs2Mgmt:1 with resDrbdShared0:1 Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave VirtualIP0#011(Started app01) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resDrbdShared0:0#011(Master app02) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resDrbdShared0:1#011(Master app01) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resOCFS:0#011(Started app02) Dec 27 13:57:15 app02 pengine: [1494]: notice: LogActions: Stop resOCFS:1#011(app01) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resJboss:0#011(Stopped unmanaged) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resJboss:1#011(Stopped unmanaged) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resDLM:0#011(Started app02) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resO2CB:0#011(Started app02) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resDLM:1#011(Started app01) Dec 27 13:57:15 app02 pengine: [1494]: info: LogActions: Leave resO2CB:1#011(Started app01) Dec 27 13:57:15 app02 crmd: [1495]: debug: s_crmd_fsa: Processing I_PE_SUCCESS: [ state=S_POLICY_ENGINE cause=C_IPC_MESSAGE origin=handle_response ] Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_LOG Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: FSA: Input I_PE_SUCCESS from handle_response() received in state S_POLICY_ENGINE Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <create_reply_adv origin="process_pe_message" t="crmd" version="3.0.6" subt="response" reference="pe_calc-dc-1356613035-752" crm_task="pe_calc" crm_sys_to="dc" crm_sys_from="pengine" crm-tgraph-in="/var/lib/pengine/pe-input-840.bz2" graph-errors="0" graph-warnings="0" config-errors="0" config-warnings="0" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <crm_xml > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <transition_graph cluster-delay="60s" stonith-timeout="60s" failed-stop-offset="INFINITY" failed-start-offset="INFINITY" batch-limit="30" transition_id="335" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <synapse id="0" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <action_set > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <rsc_op id="46" operation="stop" operation_key="resOCFS:1_stop_0" on_node="app01" on_node_uuid="app01" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <primitive id="resOCFS:1" long-id="cloneOCFS:resOCFS:1" class="ocf" provider="heartbeat" type="Filesystem" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <attributes CRM_meta_clone="1" CRM_meta_clone_max="2" CRM_meta_clone_node_max="1" CRM_meta_globally_unique="false" CRM_meta_name="stop" CRM_meta_notify="false" CRM_meta_timeout="60000" crm_feature_set="3.0.6" device="/dev/drbd/by-res/shared0" directory="/data" fstype="ocfs2" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </rsc_op> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </action_set> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <inputs > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <trigger > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <pseudo_event id="49" operation="stop" operation_key="cloneOCFS_stop_0" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </trigger> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </inputs> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </synapse> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <synapse id="1" priority="1000000" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <action_set > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <pseudo_event id="50" operation="stopped" operation_key="cloneOCFS_stopped_0" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <attributes CRM_meta_clone_max="2" CRM_meta_clone_node_max="1" CRM_meta_globally_unique="false" CRM_meta_notify="false" CRM_meta_timeout="20000" crm_feature_set="3.0.6" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </pseudo_event> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </action_set> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <inputs > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <trigger > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <rsc_op id="46" operation="stop" operation_key="resOCFS:1_stop_0" on_node="app01" on_node_uuid="app01" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </trigger> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <trigger > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <pseudo_event id="49" operation="stop" operation_key="cloneOCFS_stop_0" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </trigger> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </inputs> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </synapse> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <synapse id="2" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <action_set > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <pseudo_event id="49" operation="stop" operation_key="cloneOCFS_stop_0" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <attributes CRM_meta_clone_max="2" CRM_meta_clone_node_max="1" CRM_meta_globally_unique="false" CRM_meta_notify="false" CRM_meta_timeout="20000" crm_feature_set="3.0.6" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </pseudo_event> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </action_set> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <inputs /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </synapse> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <synapse id="3" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <action_set > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <pseudo_event id="6" operation="all_stopped" operation_key="all_stopped" > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <attributes crm_feature_set="3.0.6" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </pseudo_event> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </action_set> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <inputs > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <trigger > Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log <rsc_op id="46" operation="stop" operation_key="resOCFS:1_stop_0" on_node="app01" on_node_uuid="app01" /> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </trigger> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </inputs> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </synapse> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </transition_graph> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </crm_xml> Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: do_log </create_reply_adv> Dec 27 13:57:15 app02 crmd: [1495]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_DC_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_INTEGRATE_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_FINALIZE_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_TE_INVOKE Dec 27 13:57:15 app02 crmd: [1495]: debug: unpack_graph: Unpacked transition 335: 4 actions in 4 synapses Dec 27 13:57:15 app02 crmd: [1495]: info: do_te_invoke: Processing graph 335 (ref=pe_calc-dc-1356613035-752) derived from /var/lib/pengine/pe-input-840.bz2 Dec 27 13:57:15 app02 crmd: [1495]: debug: te_pseudo_action: Pseudo action 49 fired and confirmed Dec 27 13:57:15 app02 crmd: [1495]: debug: run_graph: ==== Transition 335 (Complete=0, Pending=0, Fired=1, Skipped=0, Incomplete=3, Source=/var/lib/pengine/pe-input-840.bz2): In-progress Dec 27 13:57:15 app02 crmd: [1495]: info: te_rsc_command: Initiating action 46: stop resOCFS:1_stop_0 on app01 Dec 27 13:57:15 app02 crmd: [1495]: debug: run_graph: ==== Transition 335 (Complete=1, Pending=1, Fired=1, Skipped=0, Incomplete=2, Source=/var/lib/pengine/pe-input-840.bz2): In-progress Dec 27 13:57:15 app02 pengine: [1494]: notice: process_pe_message: Transition 335: PEngine Input stored in: /var/lib/pengine/pe-input-840.bz2 Dec 27 13:57:15 app02 cib: [1490]: debug: cib_process_xpath: cib_query: //nvpar[@name='terminate'] does not exist Dec 27 13:57:15 app02 crmd: [1495]: debug: get_xpath_object: No match for //cib_update_result//diff-added//crm_config in /notify Dec 27 13:57:15 app02 crmd: [1495]: debug: te_update_diff: Processing diff (cib_modify): 0.356.23 -> 0.356.24 (S_TRANSITION_ENGINE) Dec 27 13:57:15 app02 crmd: [1495]: debug: match_graph_event: Action resOCFS:1_stop_0 (46) confirmed on app01 (rc=0) Dec 27 13:57:15 app02 crmd: [1495]: debug: te_pseudo_action: Pseudo action 50 fired and confirmed Dec 27 13:57:15 app02 crmd: [1495]: debug: te_pseudo_action: Pseudo action 6 fired and confirmed Dec 27 13:57:15 app02 crmd: [1495]: debug: run_graph: ==== Transition 335 (Complete=2, Pending=0, Fired=2, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-840.bz2): In-progress Dec 27 13:57:15 app02 crmd: [1495]: notice: run_graph: ==== Transition 335 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-840.bz2): Complete Dec 27 13:57:15 app02 crmd: [1495]: debug: te_graph_trigger: Transition 335 is now complete Dec 27 13:57:15 app02 crmd: [1495]: debug: notify_crmd: Processing transition completion in state S_TRANSITION_ENGINE Dec 27 13:57:15 app02 crmd: [1495]: debug: notify_crmd: Transition 335 status: done - <null> Dec 27 13:57:15 app02 crmd: [1495]: debug: s_crmd_fsa: Processing I_TE_SUCCESS: [ state=S_TRANSITION_ENGINE cause=C_FSA_INTERNAL origin=notify_crmd ] Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_LOG Dec 27 13:57:15 app02 crmd: [1495]: debug: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE Dec 27 13:57:15 app02 crmd: [1495]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Dec 27 13:57:15 app02 crmd: [1495]: debug: do_state_transition: Starting PEngine Recheck Timer Dec 27 13:57:15 app02 crmd: [1495]: debug: crm_timer_start: Started PEngine Recheck Timer (I_PE_CALC:900000ms), src=1176 Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_DC_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_INTEGRATE_TIMER_STOP Dec 27 13:57:15 app02 crmd: [1495]: debug: do_fsa_action: actions:trace: #011// A_FINALIZE_TIMER_STOP 2012/12/27 Stefan Midjich <[email protected]> > This has happened once before and was fixed by changing the order and > colocation constraints. I believe there is something wrong with this part > of the configuration so I have included the cib in this mail and hope > someone can spot the error. > > The observed events are like this. > cloneOCFS is stopped and shared volume is unmounted. > Some time passes and the resource is reprobed by pacemaker and is started. > It repeats after about 13-18 minutes of normality. > > Here is the relevant log from one such event. And the cib further down the > mail. > > Dec 27 12:27:19 app01 crmd: [1408]: debug: cancel_op: Cancelling op 34 for > resOCFS:1 (resOCFS:1:34) > Dec 27 12:27:19 app01 lrmd: [1405]: info: cancel_op: operation monitor[34] on > resOCFS:1 for client 1408, its parameters: CRM_meta_clone=[1] fstype=[ocfs2] > device=[/dev/drbd/by-res/ shared0] CRM_meta_clone_node_max=[1] > CRM_meta_clone_max=[2] CRM_meta_notify=[false] > CRM_meta_globally_unique=[false] crm_feature_set=[3.0.6] directory=[/data] > CRM_meta_name=[monitor] CRM_meta_interval=[120000] CRM_meta_timeout=[40000] > cancelled > Dec 27 12:27:19 app01 lrmd: [1405]: debug: on_msg_cancel_op: operation 34 > cancelled > Dec 27 12:27:19 app01 crmd: [1408]: debug: cancel_op: Op 34 for resOCFS:1 > (resOCFS:1:34): cancelled > Dec 27 12:27:19 app01 crmd: [1408]: debug: do_lrm_rsc_op: Performing > key=46:329:0:84c51969-1d73-4895-9798-bb05f50225aa op=resOCFS:1_stop_0 > Dec 27 12:27:19 app01 lrmd: [1405]: debug: on_msg_perform_op: add an > operation operation stop[35] on resOCFS:1 for client 1408, its parameters: > CRM_meta_clone=[1] CRM_meta_clone_node_max=[1] > CRM_meta_clone_max=[2] CRM_meta_notify=[false] crm_feature_set=[3.0.6] > CRM_meta_globally_unique=[false] CRM_meta_name=[stop] > CRM_meta_timeout=[60000] to the operation list. > Dec 27 12:27:19 app01 lrmd: [1405]: info: rsc:resOCFS:1 stop[35] (pid 2112) > Dec 27 12:27:19 app01 crmd: [1408]: info: process_lrm_event: LRM operation > resOCFS:1_monitor_120000 (call=34, status=1, cib-update=0, confirmed=true) > Cancelled > Dec 27 12:27:19 app01 crmd: [1408]: debug: update_history_cache: Appending > monitor op to history for 'resOCFS:1' > Dec 27 12:27:19 app01 Filesystem[2112]: INFO: Running stop for > /dev/drbd/by-res/shared0 on /data > Dec 27 12:27:19 app01 Filesystem[2112]: INFO: Trying to unmount /data > Dec 27 12:27:19 app01 Filesystem[2112]: INFO: unmounted /data successfully > Dec 27 12:27:19 app01 lrmd: [1405]: info: Managed resOCFS:1:stop process 2112 > exited with return code 0. > Dec 27 12:27:19 app01 lrmd: [1405]: info: operation stop[35] on resOCFS:1 for > client 1408: pid 2112 exited with return code 0 > Dec 27 12:27:19 app01 crmd: [1408]: debug: create_operation_update: > do_update_resource: Updating resouce resOCFS:1 after complete stop op > (interval=0) > Dec 27 12:27:19 app01 crmd: [1408]: info: process_lrm_event: LRM operation > resOCFS:1_stop_0 (call=35, rc=0, cib-update=65, confirmed=true) ok > Dec 27 12:27:19 app01 crmd: [1408]: debug: update_history_cache: Appending > stop op to history for 'resOCFS:1' > Dec 27 12:27:19 app01 crmd: [1408]: debug: get_xpath_object: No match for > //cib_update_result//diff-added//crm_config in /notify > > ########################################### > > primitive VirtualIP0 ocf:heartbeat:IPaddr2 \ > params ip="10.121.12.30" \ > op monitor interval="10s" \ > meta target-role="Started" > primitive resDLM ocf:pacemaker:controld > primitive resDrbdShared0 ocf:linbit:drbd \ > params drbd_resource="shared0" \ > operations $id="resDrbd-operations" \ > op monitor interval="20" role="Master" timeout="20" notify="true" \ > op monitor interval="30" role="Slave" timeout="20" notify="true" > primitive resO2CB ocf:pacemaker:o2cb > primitive resOCFS ocf:heartbeat:Filesystem \ > params device="/dev/drbd/by-res/shared0" directory="/data" > fstype="ocfs2" \ > op monitor interval="120s" timeout="40" \ > op start interval="0" timeout="60" \ > op stop interval="0" timeout="60" > group groupOcfs2Mgmt resDLM resO2CB > ms msDrbdShared0 resDrbdShared0 \ > meta resource-stickines="100" notify="true" interleave="true" > master-max="2" target-role="Started" > clone cloneOCFS resOCFS \ > meta interleave="true" ordered="true" target-role="Started" > clone cloneOcfs2Mgmt groupOcfs2Mgmt \ > meta interleave="true" target-role="Started" > location locVirtualIP0 VirtualIP0 9001: app01 > colocation colDRBD inf: cloneOcfs2Mgmt msDrbdShared0:Master > colocation colOcfs2 inf: cloneOCFS cloneOcfs2Mgmt > order ordDRBD inf: msDrbdShared0:promote cloneOcfs2Mgmt:start > order ordOcfs2 inf: cloneOcfs2Mgmt:start cloneOCFS:start > property $id="cib-bootstrap-options" \ > dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \ > cluster-infrastructure="openais" \ > expected-quorum-votes="2" \ > no-quorum-policy="ignore" \ > last-lrm-refresh="1356604931" > rsc_defaults $id="rsc-options" \ > resource-stickiness="0" > op_defaults $id="op-options" \ > timeout="20s" > > > > -- > Hälsningar / Greetings > > Stefan Midjich > [De omnibus dubitandum] > -- Hälsningar / Greetings Stefan Midjich [De omnibus dubitandum] _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
