Hi, I read at http://www.linux-ha.org/HeartbeatResourceAgent that "After repeated stop failures, we will do a status on the resource. If the status reports that the resource is still running, then we will reboot the machine to make sure things are really stopped."
I have a 2 node cluster and the nodes are connected per Ata-over-Ethernet to some Filesystems. Each resource depends on pingd and will be stopped if there is no connectivity. So when I disconnect node mrbrown from the network I'll get a split-brain condition. Because of the pingd constraint mrbrown tries to stop the Filesystem-resource. This will timeout or produce errors because the Filesystem is not reachable. But mrbrown doesn't reboot as it should. It case of ignores that error?!? Did I made a mistake? Btw. there is no real stonith device at the moment, just ssh and this will not work without network. You can find some output of mrbrown in close. Any help is really appreciated.. Cheers, Holger
Feb 28 19:08:23 mrbrown crmd: [9179]: info: do_dc_join_ack: join-1: Updating node state to member for mrbrown Feb 28 19:08:23 mrbrown tengine: [18466]: info: process_graph_event: Action pingd:0_monitor_0 initiated by a different transitioner Feb 28 19:08:23 mrbrown tengine: [18466]: info: process_graph_event: Action openvz:0_monitor_0 initiated by a different transitioner Feb 28 19:08:23 mrbrown tengine: [18466]: info: process_graph_event: Action Filesystem-ve101_monitor_0 initiated by a different transitioner Feb 28 19:08:23 mrbrown crmd: [9179]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Feb 28 19:08:23 mrbrown crmd: [9179]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Feb 28 19:08:23 mrbrown pengine: [18467]: info: determine_online_status: Node mrbrown is online Feb 28 19:08:23 mrbrown pengine: [18467]: notice: clone_print: Clone Set: openvz-clone Feb 28 19:08:23 mrbrown pengine: [18467]: notice: native_print: openvz:0^I(lsb:vz):^IStopped Feb 28 19:08:23 mrbrown pengine: [18467]: notice: native_print: openvz:1^I(lsb:vz):^IStarted mrbrown Feb 28 19:08:23 mrbrown pengine: [18467]: notice: group_print: Resource Group: ve101 Feb 28 19:08:23 mrbrown pengine: [18467]: notice: native_print: Filesystem-ve101^I(heartbeat::ocf:Filesystem):^IStarted mrbrown Feb 28 19:08:23 mrbrown pengine: [18467]: notice: clone_print: Clone Set: pingd-clone Feb 28 19:08:23 mrbrown pengine: [18467]: notice: native_print: pingd:0^I(heartbeat::ocf:pingd):^IStopped Feb 28 19:08:23 mrbrown pengine: [18467]: notice: native_print: pingd:1^I(heartbeat::ocf:pingd):^IStarted mrbrown Feb 28 19:08:23 mrbrown pengine: [18467]: WARN: native_color: Resource openvz:0 cannot run anywhere Feb 28 19:08:23 mrbrown pengine: [18467]: notice: NoRoleChange: Leave resource openvz:1^I(mrbrown) Feb 28 19:08:23 mrbrown pengine: [18467]: WARN: native_color: Resource Filesystem-ve101 cannot run anywhere Feb 28 19:08:23 mrbrown pengine: [18467]: notice: StopRsc: mrbrown^IStop Filesystem-ve101 Feb 28 19:08:23 mrbrown pengine: [18467]: WARN: native_color: Resource pingd:0 cannot run anywhere Feb 28 19:08:23 mrbrown pengine: [18467]: notice: NoRoleChange: Leave resource pingd:1^I(mrbrown) Feb 28 19:08:23 mrbrown crmd: [9179]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] Feb 28 19:08:23 mrbrown tengine: [18466]: info: unpack_graph: Unpacked transition 0: 4 actions in 4 synapses Feb 28 19:08:23 mrbrown tengine: [18466]: info: te_pseudo_action: Pseudo action 15 fired and confirmed Feb 28 19:08:23 mrbrown tengine: [18466]: info: send_rsc_command: Initiating action 12: Filesystem-ve101_stop_0 on mrbrown Feb 28 19:08:23 mrbrown crmd: [9179]: info: do_lrm_rsc_op: Performing op=Filesystem-ve101_stop_0 key=12:0:de878f16-1f38-4454-853a-1c93f0e86b47) Feb 28 19:08:23 mrbrown lrmd: [9176]: info: rsc:Filesystem-ve101: stop Feb 28 19:08:23 mrbrown crmd: [9179]: info: process_lrm_event: LRM operation Filesystem-ve101_monitor_10000 (call=13, rc=-2) Cancelled Feb 28 19:08:23 mrbrown Filesystem[18469]: [18499]: INFO: Running stop for /dev/etherd/e101.0 on /vz/101 Feb 28 19:08:23 mrbrown Filesystem[18469]: [18509]: INFO: Trying to unmount /vz/101 Feb 28 19:08:43 mrbrown tengine: [18466]: WARN: action_timer_callback: Timer popped (abort_level=0, complete=false) Feb 28 19:08:43 mrbrown tengine: [18466]: WARN: print_elem: Action missed its timeout[Action 12]: In-flight (id: Filesystem-ve101_stop_0, loc: mrbrown, priority: 0) Feb 28 19:08:43 mrbrown lrmd: [9176]: WARN: Filesystem-ve101:stop process (PID 18469) timed out (try 1). Killing with signal SIGTERM (15). Feb 28 19:08:43 mrbrown lrmd: [9176]: WARN: operation stop[14] on ocf::Filesystem::Filesystem-ve101 for client 9179, its parameters: directory=[/vz/101] fstype=[ext3] device=[/dev/etherd/e101.0] CRM_meta_timeout=[20000] crm_feature_set=[2. 1] options=[noatime,ro] : pid [18469] timed out Feb 28 19:08:43 mrbrown crmd: [9179]: ERROR: process_lrm_event: LRM operation Filesystem-ve101_stop_0 (14) Timed Out (timeout=20000ms) Feb 28 19:08:43 mrbrown tengine: [18466]: info: status_from_rc: Re-mapping op status to LRM_OP_ERROR for rc=-2 Feb 28 19:08:43 mrbrown tengine: [18466]: WARN: status_from_rc: Action stop on mrbrown failed (target: <null> vs. rc: -2): Error Feb 28 19:08:43 mrbrown tengine: [18466]: WARN: update_failcount: Updating failcount for Filesystem-ve101 on 08d79e6d-38ea-46ca-bebb-c2bffca1045c after failed stop: rc=-2 Feb 28 19:08:43 mrbrown tengine: [18466]: info: update_abort_priority: Abort priority upgraded to 1 Feb 28 19:08:43 mrbrown tengine: [18466]: info: update_abort_priority: Abort action 0 superceeded by 2 Feb 28 19:08:43 mrbrown tengine: [18466]: info: match_graph_event: Action Filesystem-ve101_stop_0 (12) confirmed on mrbrown (rc=4) Feb 28 19:08:43 mrbrown tengine: [18466]: info: extract_event: Aborting on transient_attributes changes for 08d79e6d-38ea-46ca-bebb-c2bffca1045c Feb 28 19:08:43 mrbrown tengine: [18466]: info: update_abort_priority: Abort priority upgraded to 1000000 Feb 28 19:08:43 mrbrown tengine: [18466]: info: update_abort_priority: 'Event failed' abort superceeded Feb 28 19:08:43 mrbrown tengine: [18466]: info: run_graph: ==================================================== Feb 28 19:08:43 mrbrown tengine: [18466]: notice: run_graph: Transition 0: (Complete=2, Pending=0, Fired=0, Skipped=2, Incomplete=0) Feb 28 19:08:43 mrbrown crmd: [9179]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ] Feb 28 19:08:43 mrbrown crmd: [9179]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
