Sorry, I think no attachment can be made here. So pasting the log instead
ccm[13871]: 1980/02/01_02:41:41 info: client (pid=13872) removed from ccm
heartbeat[13863]: 1980/02/01_02:41:41 info: killing
/usr/lib/heartbeat/ccm process group 13871 with signal 15
ccm[13871]: 1980/02/01_02:41:42 info: received SIGTERM, going to shut down
heartbeat[13863]: 1980/02/01_02:41:43 info: killing HBFIFO process
13867 with signal 15
heartbeat[13863]: 1980/02/01_02:41:43 info: killing HBWRITE process
13868 with signal 15
heartbeat[13863]: 1980/02/01_02:41:43 info: killing HBREAD process
13869 with signal 15
heartbeat[13863]: 1980/02/01_02:41:43 info: Core process 13869 exited.
3 remaining
heartbeat[13863]: 1980/02/01_02:41:43 info: Core process 13868 exited.
2 remaining
heartbeat[13863]: 1980/02/01_02:41:43 info: Core process 13867 exited.
1 remaining
heartbeat[13863]: 1980/02/01_02:41:43 info: clusternode2 Heartbeat
shutdown complete.
heartbeat[22132]: 1980/02/03_00:19:28 info: Version 2 support: on
heartbeat[22132]: 1980/02/03_00:19:28 WARN: File /etc/ha.d/haresources exists.
heartbeat[22132]: 1980/02/03_00:19:28 WARN: This file is not used
because crm is enabled
heartbeat[22132]: 1980/02/03_00:19:28 WARN: Logging daemon is disabled
--enabling logging daemon is recommended
heartbeat[22132]: 1980/02/03_00:19:28 info: **************************
heartbeat[22132]: 1980/02/03_00:19:28 info: Configuration validated.
Starting heartbeat 2.1.3
heartbeat[22133]: 1980/02/03_00:19:28 info: heartbeat: version 2.1.3
heartbeat[22133]: 1980/02/03_00:19:28 info: Heartbeat generation: 1224679739
heartbeat[22133]: 1980/02/03_00:19:28 info: glib: UDP Broadcast
heartbeat started on port 694 (694) interface eth0
heartbeat[22133]: 1980/02/03_00:19:28 info: glib: UDP Broadcast
heartbeat closed on port 694 interface eth0 - Status: 1
heartbeat[22133]: 1980/02/03_00:19:28 info: G_main_add_TriggerHandler:
Added signal manual handler
heartbeat[22133]: 1980/02/03_00:19:28 info: G_main_add_TriggerHandler:
Added signal manual handler
heartbeat[22133]: 1980/02/03_00:19:28 info: G_main_add_SignalHandler:
Added signal handler for signal 17
heartbeat[22133]: 1980/02/03_00:19:28 info: Local status now set to: 'up'
heartbeat[22133]: 1980/02/03_00:19:29 info: Link clusternode2:eth0 up.
heartbeat[22133]: 1980/02/03_00:19:39 WARN: node clusternode1: is dead
heartbeat[22133]: 1980/02/03_00:19:39 info: Comm_now_up(): updating
status to active
heartbeat[22133]: 1980/02/03_00:19:39 info: Local status now set to: 'active'
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/ccm" (498,496)
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/cib" (498,496)
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/lrmd -r" (0,0)
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/stonithd" (0,0)
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/attrd" (498,496)
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/crmd" (498,496)
heartbeat[22133]: 1980/02/03_00:19:39 info: Starting child client
"/usr/lib/heartbeat/mgmtd -v" (0,0)
heartbeat[22140]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/ccm" as uid 498 gid 496 (pid 22140)
heartbeat[22141]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/cib" as uid 498 gid 496 (pid 22141)
cib[22141]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 15
cib[22141]: 1980/02/03_00:19:39 info: G_main_add_TriggerHandler: Added
signal manual handler
cib[22141]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 17
cib[22141]: 1980/02/03_00:19:39 info: main: Retrieval of a per-action
CIB: disabled
cib[22141]: 1980/02/03_00:19:39 info: retrieveCib: Reading cluster
configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Expected: resources
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error parsing token:
Mismatching close tag
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error at or before:
</configuration>
</
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error parsing token:
error parsing child
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error at or before:
<resources>
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error parsing token:
error parsing child
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error at or before:
<configuration>
cib[22141]: 1980/02/03_00:19:39 ERROR: retrieveCib:
/var/lib/heartbeat/crm/cib.xml exists but does NOT contain valid XML.
heartbeat[22144]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/attrd" as uid 498 gid 496 (pid 22144)
heartbeat[22145]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/crmd" as uid 498 gid 496 (pid 22145)
attrd[22144]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 15
attrd[22144]: 1980/02/03_00:19:39 info: register_with_ha: Hostname: clusternode2
crmd[22145]: 1980/02/03_00:19:39 info: main: CRM Hg Version: node:
552305612591183b1628baa5bc6e903e0f1e26a3
crmd[22145]: 1980/02/03_00:19:39 info: crmd_init: Starting crmd
crmd[22145]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 15
crmd[22145]: 1980/02/03_00:19:39 info: G_main_add_TriggerHandler:
Added signal manual handler
heartbeat[22143]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 22143)
heartbeat[22146]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 22146)
mgmtd[22146]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 15
mgmtd[22146]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 10
mgmtd[22146]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 12
mgmtd[22146]: 1980/02/03_00:19:39 WARN: lrm_signon: can not initiate connection
mgmtd[22146]: 1980/02/03_00:19:39 info: login to lrm: 0, ret:0
crmd[22145]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 17
cib[22141]: 1980/02/03_00:19:39 WARN: retrieveCib: Continuing but
/var/lib/heartbeat/crm/cib.xml will NOT used.
cib[22141]: 1980/02/03_00:19:39 WARN: readCibXmlFile: Primary
configuration corrupt or unusable, trying backup...
cib[22141]: 1980/02/03_00:19:39 info: retrieveCib: Reading cluster
configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest:
/var/lib/heartbeat/crm/cib.xml.sig.last)
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Expected: resources
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error parsing token:
Mismatching close tag
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error at or before:
</configuration>
</
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error parsing token:
error parsing child
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error at or before:
<resources>
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error parsing token:
error parsing child
cib[22141]: 1980/02/03_00:19:39 ERROR: parse_xml: Error at or before:
<configuration>
heartbeat[22142]: 1980/02/03_00:19:39 info: Starting
"/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 22142)
lrmd[22142]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 15
lrmd[22142]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 17
lrmd[22142]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 10
lrmd[22142]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler: Added
signal handler for signal 12
lrmd[22142]: 1980/02/03_00:19:39 info: Started.
ccm[22140]: 1980/02/03_00:19:39 info: Hostname: clusternode2
cib[22141]: 1980/02/03_00:19:39 ERROR: retrieveCib:
/var/lib/heartbeat/crm/cib.xml.last exists but does NOT contain valid
XML.
cib[22141]: 1980/02/03_00:19:39 WARN: retrieveCib: Continuing but
/var/lib/heartbeat/crm/cib.xml.last will NOT used.
cib[22141]: 1980/02/03_00:19:39 WARN: readCibXmlFile: Continuing with
an empty configuration.
attrd[22144]: 1980/02/03_00:19:39 info: register_with_ha: UUID:
d354acf4-568d-4583-9fe9-72eabb2835b1
stonithd[22143]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 10
cib[22141]: 1980/02/03_00:19:39 WARN: readCibXmlFile: No value for
admin_epoch was specified in the configuration.
stonithd[22143]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 12
stonithd[22143]: 1980/02/03_00:19:39 info: Signing in with heartbeat.
cib[22141]: 1980/02/03_00:19:39 WARN: readCibXmlFile: The reccomended
course of action is to shutdown, run crm_verify and fix any errors it
reports.
cib[22141]: 1980/02/03_00:19:39 WARN: readCibXmlFile: We will default
to zero and continue but may get confused about which configuration to
use if multiple nodes are powered up at the same time.
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <cib generated="true" admin_epoch="0"
epoch="0" num_updates="0" have_quorum="false">
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <configuration>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <crm_config/>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <nodes/>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <resources/>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <constraints/>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] </configuration>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] <status/>
cib[22141]: 1980/02/03_00:19:39 info: log_data_element:
readCibXmlFile: [on-disk] </cib>
cib[22141]: 1980/02/03_00:19:39 notice: readCibXmlFile: Enabling DTD
validation on the existing (sane) configuration
cib[22141]: 1980/02/03_00:19:39 info: startCib: CIB Initialization
completed successfully
cib[22141]: 1980/02/03_00:19:39 info: cib_register_ha: Signing in with Heartbeat
cib[22141]: 1980/02/03_00:19:39 info: cib_register_ha: FSA Hostname:
clusternode2
cib[22141]: 1980/02/03_00:19:39 info: ccm_connect: Registering with CCM...
cib[22141]: 1980/02/03_00:19:39 WARN: ccm_connect: CCM Activation failed
cib[22141]: 1980/02/03_00:19:39 WARN: ccm_connect: CCM Connection
failed 1 times (30 max)
stonithd[22143]: 1980/02/03_00:19:39 notice:
/usr/lib/heartbeat/stonithd start up successfully.
stonithd[22143]: 1980/02/03_00:19:39 info: G_main_add_SignalHandler:
Added signal handler for signal 17
mgmtd[22146]: 1980/02/03_00:19:40 info: init_crm
cib[22141]: 1980/02/03_00:19:42 info: ccm_connect: Registering with CCM...
cib[22141]: 1980/02/03_00:19:42 WARN: ccm_connect: CCM Activation failed
cib[22141]: 1980/02/03_00:19:42 WARN: ccm_connect: CCM Connection
failed 2 times (30 max)
ccm[22140]: 1980/02/03_00:19:42 info: Break tie for 2 nodes cluster
ccm[22140]: 1980/02/03_00:19:42 info: G_main_add_SignalHandler: Added
signal handler for signal 15
cib[22141]: 1980/02/03_00:19:45 info: ccm_connect: Registering with CCM...
cib[22141]: 1980/02/03_00:19:45 info: cib_init: Starting cib mainloop
cib[22141]: 1980/02/03_00:19:45 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[22141]: 1980/02/03_00:19:45 info: mem_handle_event: instance=1,
nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
cib[22141]: 1980/02/03_00:19:45 info: cib_ccm_msg_callback: PEER: clusternode2
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Expected: resources
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Error parsing token:
Mismatching close tag
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Error at or before:
</configuration>
</
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Error parsing token:
error parsing child
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Error at or before:
<resources>
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Error parsing token:
error parsing child
cib[22148]: 1980/02/03_00:19:45 ERROR: parse_xml: Error at or before:
<configuration>
cib[22141]: 1980/02/03_00:19:45 info: cib_null_callback: Setting
cib_diff_notify callbacks for mgmtd: on
cib[22141]: 1980/02/03_00:19:46 info: cib_client_status_callback:
Status update: Client clusternode2/cib now has status [join]
cib[22141]: 1980/02/03_00:19:46 info: cib_client_status_callback:
Status update: Client clusternode2/cib now has status [online]
mgmtd[22146]: 1980/02/03_00:19:46 info: Started.
cib[22148]: 1980/02/03_00:19:46 ERROR: validate_cib_digest: Digest
comparision failed: expected 1ed79288dfd53bdc1c316e24ab0c1940
(/var/lib/heartbeat/crm/cib.xml.sig), calculated (null)
cib[22148]: 1980/02/03_00:19:46 ERROR: write_cib_contents:
/var/lib/heartbeat/crm/cib.xml was manually modified while Heartbeat
was active!
cib[22141]: 1980/02/03_00:19:46 ERROR: cib_diskwrite_complete: Disk
write failed: status=256, signo=0, exitcode=1
cib[22141]: 1980/02/03_00:19:46 ERROR: cib_diskwrite_complete:
Disabling disk writes after write failure
cib[22141]: 1980/02/03_00:19:46 info: cib_null_callback: Setting
cib_refresh_notify callbacks for crmd: on
crmd[22145]: 1980/02/03_00:19:46 info: do_cib_control: CIB connection
established
crmd[22145]: 1980/02/03_00:19:46 info: register_with_ha: Hostname: clusternode2
crmd[22145]: 1980/02/03_00:19:46 info: register_with_ha: UUID:
d354acf4-568d-4583-9fe9-72eabb2835b1
crmd[22145]: 1980/02/03_00:19:46 info: populate_cib_nodes: Requesting
the list of configured nodes
crmd[22145]: 1980/02/03_00:19:47 notice: populate_cib_nodes: Node:
clusternode2 (uuid: d354acf4-568d-4583-9fe9-72eabb2835b1)
crmd[22145]: 1980/02/03_00:19:48 WARN: get_uuid: Could not calculate
UUID for clusternode1
crmd[22145]: 1980/02/03_00:19:48 WARN: populate_cib_nodes: Node
clusternode1: no uuid found
crmd[22145]: 1980/02/03_00:19:48 info: do_ha_control: Connected to Heartbeat
crmd[22145]: 1980/02/03_00:19:48 info: do_ccm_control: CCM connection
established... waiting for first callback
crmd[22145]: 1980/02/03_00:19:48 info: do_started: Delaying start, CCM
(0000000000100000) not connected
crmd[22145]: 1980/02/03_00:19:48 info: crmd_init: Starting crmd's mainloop
crmd[22145]: 1980/02/03_00:19:48 notice: crmd_client_status_callback:
Status update: Client clusternode2/crmd now has status [online]
crmd[22145]: 1980/02/03_00:19:48 notice: crmd_client_status_callback:
Status update: Client clusternode2/crmd now has status [online]
crmd[22145]: 1980/02/03_00:19:48 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
crmd[22145]: 1980/02/03_00:19:48 info: mem_handle_event: instance=1,
nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
crmd[22145]: 1980/02/03_00:19:48 info: crmd_ccm_msg_callback: Quorum
(re)attained after event=NEW MEMBERSHIP (id=1)
crmd[22145]: 1980/02/03_00:19:48 info: ccm_event_detail: NEW
MEMBERSHIP: trans=1, nodes=1, new=1, lost=0 n_idx=0, new_idx=0,
old_idx=3
crmd[22145]: 1980/02/03_00:19:48 info: ccm_event_detail: CURRENT:
clusternode2 [nodeid=1, born=1]
crmd[22145]: 1980/02/03_00:19:48 info: ccm_event_detail: NEW:
clusternode2 [nodeid=1, born=1]
crmd[22145]: 1980/02/03_00:19:48 info: do_started: The local CRM is operational
crmd[22145]: 1980/02/03_00:19:48 info: do_state_transition: State
transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_CCM_CALLBACK origin=do_started ]
attrd[22144]: 1980/02/03_00:19:51 info: main: Starting mainloop...
crmd[22145]: 1980/02/03_00:19:59 info: crm_timer_popped: Election
Trigger (I_DC_TIMEOUT) just popped!
crmd[22145]: 1980/02/03_00:19:59 WARN: do_log: [[FSA]] Input
I_DC_TIMEOUT from crm_timer_popped() received in state (S_PENDING)
crmd[22145]: 1980/02/03_00:19:59 info: do_state_transition: State
transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
cause=C_TIMER_POPPED origin=crm_timer_popped ]
crmd[22145]: 1980/02/03_00:19:59 info: do_election_count_vote: Updated
voted hash for clusternode2 to vote
crmd[22145]: 1980/02/03_00:19:59 info: do_election_count_vote:
Election ignore: our vote (clusternode2)
crmd[22145]: 1980/02/03_00:19:59 info: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
crmd[22145]: 1980/02/03_00:19:59 info: start_subsystem: Starting
sub-system "tengine"
crmd[22145]: 1980/02/03_00:19:59 info: start_subsystem: Starting
sub-system "pengine"
pengine[22151]: 1980/02/03_00:19:59 info: G_main_add_SignalHandler:
Added signal handler for signal 15
pengine[22151]: 1980/02/03_00:19:59 info: pe_init: Starting pengine
crmd[22145]: 1980/02/03_00:19:59 info: do_dc_takeover: Taking over DC
status for this partition
tengine[22150]: 1980/02/03_00:19:59 info: G_main_add_SignalHandler:
Added signal handler for signal 15
tengine[22150]: 1980/02/03_00:19:59 info: G_main_add_TriggerHandler:
Added signal manual handler
tengine[22150]: 1980/02/03_00:19:59 info: G_main_add_TriggerHandler:
Added signal manual handler
cib[22141]: 1980/02/03_00:19:59 info: cib_process_readwrite: We are
now in R/W mode
cib[22141]: 1980/02/03_00:19:59 info: revision_check: Updating CIB
revision to 2.0
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: -
<cib epoch="0"/>
crmd[22145]: 1980/02/03_00:19:59 info: join_make_offer: Making join
offers based on membership 1
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
<cib epoch="1">
crmd[22145]: 1980/02/03_00:19:59 info: do_dc_join_offer_all: join-1:
Waiting on 1 outstanding join acks
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
<configuration>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
<crm_config>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
<cluster_property_set id="cib-bootstrap-options">
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
<attributes>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
<nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
</attributes>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
</cluster_property_set>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
</crm_config>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: +
</configuration>
cib[22141]: 1980/02/03_00:19:59 info: log_data_element: cib:diff: + </cib>
cib[22141]: 1980/02/03_00:19:59 info: cib_null_callback: Setting
cib_diff_notify callbacks for tengine: on
tengine[22150]: 1980/02/03_00:19:59 info: te_init: Registering TE
UUID: 4ae1b2f1-1238-438b-99d5-1a6d8f2c2915
tengine[22150]: 1980/02/03_00:19:59 info: set_graph_functions: Setting
custom graph functions
tengine[22150]: 1980/02/03_00:19:59 info: unpack_graph: Unpacked
transition -1: 0 actions in 0 synapses
tengine[22150]: 1980/02/03_00:19:59 info: te_init: Starting tengine
tengine[22150]: 1980/02/03_00:19:59 info: te_connect_stonith:
Attempting connection to fencing daemon...
crmd[22145]: 1980/02/03_00:20:00 info: update_dc: Set DC to clusternode2 (2.0)
crmd[22145]: 1980/02/03_00:20:00 info: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
crmd[22145]: 1980/02/03_00:20:00 info: do_state_transition: All 1
cluster nodes responded to the join offer.
crmd[22145]: 1980/02/03_00:20:00 info: update_attrd: Connecting to attrd...
cib[22141]: 1980/02/03_00:20:00 info: sync_our_cib: Syncing CIB to all peers
attrd[22144]: 1980/02/03_00:20:00 info: attrd_local_callback: Sending
full refresh
crmd[22145]: 1980/02/03_00:20:00 info: update_dc: Set DC to clusternode2 (2.0)
tengine[22150]: 1980/02/03_00:20:00 info: te_connect_stonith: Connected
crmd[22145]: 1980/02/03_00:20:01 info: do_dc_join_ack: join-1:
Updating node state to member for clusternode2
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: All 1
cluster nodes are eligible to run resources.
tengine[22150]: 1980/02/03_00:20:01 info: update_abort_priority: Abort
priority upgraded to 1000000
tengine[22150]: 1980/02/03_00:20:01 info: update_abort_priority: 'DC
Takeover' abort superceeded
pengine[22151]: 1980/02/03_00:20:01 info: determine_online_status:
Node clusternode2 is online
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=route_message ]
tengine[22150]: 1980/02/03_00:20:01 info: unpack_graph: Unpacked
transition 0: 1 actions in 1 synapses
tengine[22150]: 1980/02/03_00:20:01 info: send_rsc_command: Initiating
action 2: probe_complete on clusternode2
tengine[22150]: 1980/02/03_00:20:01 info: run_graph: Transition 0:
(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0)
tengine[22150]: 1980/02/03_00:20:01 info: notify_crmd: Transition 0
status: te_complete - <null>
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_IPC_MESSAGE origin=route_message ]
tengine[22150]: 1980/02/03_00:20:01 info: extract_event: Aborting on
transient_attributes changes for d354acf4-568d-4583-9fe9-72eabb2835b1
tengine[22150]: 1980/02/03_00:20:01 info: update_abort_priority: Abort
priority upgraded to 1000000
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_IPC_MESSAGE origin=route_message ]
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: All 1
cluster nodes are eligible to run resources.
pengine[22151]: 1980/02/03_00:20:01 info: process_pe_message:
Transition 0: PEngine Input stored in:
/var/lib/heartbeat/pengine/pe-input-9.bz2
pengine[22151]: 1980/02/03_00:20:01 info: determine_online_status:
Node clusternode2 is online
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=route_message ]
pengine[22151]: 1980/02/03_00:20:01 info: process_pe_message:
Transition 1: PEngine Input stored in:
/var/lib/heartbeat/pengine/pe-input-10.bz2
tengine[22150]: 1980/02/03_00:20:01 info: unpack_graph: Unpacked
transition 1: 0 actions in 0 synapses
tengine[22150]: 1980/02/03_00:20:01 info: run_graph: Transition 1:
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0)
tengine[22150]: 1980/02/03_00:20:01 info: notify_crmd: Transition 1
status: te_complete - <null>
crmd[22145]: 1980/02/03_00:20:01 info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_IPC_MESSAGE origin=route_message ]
On Wed, Dec 17, 2008 at 9:26 AM, Tanveer Chowdhury
<[email protected]> wrote:
> Attached is the ha log from /var/log/ha-log when I start the heartbeat
> service.
> below is the processlist created in linux
> #ps afx
> .......
> .........
> 22111 pts/0 S 0:00 ha_logd: read process
> 22112 pts/0 S 0:00 \_ ha_logd: write process
> 22133 ? SLs 0:00 heartbeat: master control process
> 22136 ? SL 0:00 \_ heartbeat: FIFO reader
> 22137 ? SL 0:00 \_ heartbeat: write: bcast eth0
> 22138 ? SL 0:00 \_ heartbeat: read: bcast eth0
> 22140 ? S 0:00 \_ /usr/lib/heartbeat/ccm
> 22141 ? S 0:00 \_ /usr/lib/heartbeat/cib
> 22142 ? S 0:00 \_ /usr/lib/heartbeat/lrmd -r
> 22143 ? SL 0:00 \_ /usr/lib/heartbeat/stonithd
> 22144 ? S 0:00 \_ /usr/lib/heartbeat/attrd
> 22145 ? S 0:00 \_ /usr/lib/heartbeat/crmd
> 22150 ? S 0:00 | \_ /usr/lib/heartbeat/tengine
> 22151 ? S 0:00 | \_ /usr/lib/heartbeat/pengine
> 22146 ? S 0:00 \_ /usr/lib/heartbeat/mgmtd -v
>
>
> On Mon, Dec 15, 2008 at 2:20 PM, Andrew Beekhof <[email protected]> wrote:
>> Logs?
>>
>> On Mon, Dec 15, 2008 at 05:03, Tanveer Chowdhury
>> <[email protected]> wrote:
>>> Hi,
>>> I have configured the below in RHEL but it doesn't start the Virtual
>>> IP address or even the services.
>>> Below is the settings I took. Most probably the cib.xml file is not in
>>> right format.
>>>
>>> Thanks in advance.
>>>
>>> # cat /etc/ha.d/ha.cf
>>>
>>> debugfile /var/log/ha-debug
>>> logfile /var/log/ha-log
>>> logfacility local0
>>> keepalive 1
>>> deadtime 5
>>> warntime 3
>>> initdead 10
>>> udpport 694
>>> bcast eth0
>>> auto_failback on
>>> node clusternode1
>>> node clusternode2
>>> crm on
>>>
>>>
>>> # vi /etc/ha.d/haresources
>>> clusternode1 10.10.4.xxx httpd
>>>
>>>
>>> # vi /etc/ha.d/authkeys
>>> auth 1
>>> 1 crc
>>>
>>> # ll /etc/init.d/apache
>>> lrwxrwxrwx 1 root root 31 Feb 1 00:20 /etc/init.d/apache ->
>>> /usr/local/apache/bin/apachectl
>>> # ll /etc/init.d/httpd
>>> lrwxrwxrwx 1 root root 31 Oct 22 2008 /etc/init.d/httpd ->
>>> /usr/local/apache/bin/apachectl
>>>
>>> # rpm -qa | grep heartbeat
>>> heartbeat-pils-2.1.3-3.el5.centos
>>> heartbeat-stonith-2.1.3-3.el5.centos
>>> heartbeat-2.1.3-3.el5.centos
>>>
>>> # uname -a
>>> Linux clusternode2 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:02 EDT 2007
>>> i686 i686 i386 GNU/Linux
>>>
>>>
>>> # cat /etc/issue
>>> Red Hat Enterprise Linux Server release 5.1 (Tikanga)
>>>
>>> This cib.xml file was auto generated after I started heartbeat service.
>>>
>>> # vi /var/lib/heartbeat/crm/cib.xml
>>>
>>> <cib generated="true" admin_epoch="0" epoch="1" num_updates="1"
>>> have_quorum="true" ignore_dtd="false" ccm_transition="1" num_peers="1"
>>> cib_feature_revision="2.0" cib-last-written="Fri Dec 15 00:15:48
>>> 2008">
>>> <configuration>
>>> <crm_config>
>>> <cluster_property_set id="cib-bootstrap-options">
>>> <attributes>
>>> <nvpair id="cib-bootstrap-options-dc-version"
>>> name="dc-version" value="2.1.3-node:
>>> 552305612591183b1628baa5bc6e903e0f1e26a3"/>
>>> </attributes>
>>> </cluster_property_set>
>>> </crm_config>
>>> <nodes>
>>> <node id="d354acf4-568d-4583-9fe9-72eabb2835b1"
>>> uname="clusternode2" type="normal"/>
>>> </nodes>
>>> <resources/>
>>> <constraints/>
>>> </configuration>
>>> </cib>
>>>
>>> Then I added these lines with it as follows
>>> <resources>
>>> <group id="apache_group">
>>> <primitive id="ip_resource_1" class="ocf" type="IPaddr"
>>> provider="heartbeat">
>>> <instance_attributes>
>>> <attributes>
>>> <nvpair name="ip" value="10.10.4.xxx"/>
>>> </attributes>
>>> </instance_attributes>
>>> </primitive>
>>> <primitive id="apache" class="heartbeat" type="apache"/>
>>> </group>
>>>
>>> So it became like
>>> <cib generated="true" admin_epoch="0" epoch="1" num_updates="1"
>>> have_quorum="true" ignore_dtd="false" ccm_transition="1" num_peers="1"
>>> cib_feature_revision="2.0" cib-last-written="Fri Dec 15 00:15:48
>>> 2008">
>>> <configuration>
>>> <crm_config>
>>> <cluster_property_set id="cib-bootstrap-options">
>>> <attributes>
>>> <nvpair id="cib-bootstrap-options-dc-version"
>>> name="dc-version" value="2.1.3-node:
>>> 552305612591183b1628baa5bc6e903e0f1e26a3"/>
>>> </attributes>
>>> </cluster_property_set>
>>> </crm_config>
>>> <nodes>
>>> <node id="d354acf4-568d-4583-9fe9-72eabb2835b1"
>>> uname="clusternode2" type="normal"/>
>>> </nodes>
>>> <resources>
>>> <group id="apache_group">
>>> <primitive id="ip_resource_1" class="ocf" type="IPaddr"
>>> provider="heartbeat">
>>> <instance_attributes>
>>> <attributes>
>>> <nvpair name="ip" value="10.10.4.xxx"/>
>>> </attributes>
>>> </instance_attributes>
>>> </primitive>
>>> <primitive id="apache" class="heartbeat" type="apache"/>
>>> </group>
>>> <resources/>
>>> <constraints/>
>>> </configuration>
>>> </cib>
>>> _______________________________________________________
>>> Linux-HA-Dev: [email protected]
>>> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
>>> Home Page: http://linux-ha.org/
>>>
>> _______________________________________________________
>> Linux-HA-Dev: [email protected]
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
>> Home Page: http://linux-ha.org/
>>
>
_______________________________________________________
Linux-HA-Dev: [email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
