heartbeat[15600]: 2007/08/08_08:54:18 info: Heartbeat restart on node linux6
heartbeat[15600]: 2007/08/08_08:54:18 info: Link linux6:eth1 up.
heartbeat[15600]: 2007/08/08_08:54:18 info: Status update for node linux6: status init
heartbeat[15600]: 2007/08/08_08:54:18 debug: Status seqno: 1 msgtime: 1186534457
heartbeat[15600]: 2007/08/08_08:54:18 debug: StartNextRemoteRscReq() - calling hook
heartbeat[15600]: 2007/08/08_08:54:18 debug: notify_world: invoking harc: OLD status: active
heartbeat[15600]: 2007/08/08_08:54:18 debug: Process [status] started pid 21247
heartbeat[15600]: 2007/08/08_08:54:18 debug: Starting notify process [status]
heartbeat[15600]: 2007/08/08_08:54:18 info: Link linux6:eth2 up.
heartbeat[15600]: 2007/08/08_08:54:18 info: Status update for node linux6: status up
heartbeat[15600]: 2007/08/08_08:54:18 debug: Status seqno: 2 msgtime: 1186534457
heartbeat[15600]: 2007/08/08_08:54:18 debug: StartNextRemoteRscReq(): child count 1
heartbeat[21247]: 2007/08/08_08:54:18 debug: notify_world: setting SIGCHLD Handler to SIG_DFL
heartbeat[21247]: 2007/08/08_08:54:18 debug: notify_world: Running harc status
harc[21247]:	2007/08/08_08:54:18 info: Running /usr/local/etc/ha.d/rc.d/status status
heartbeat[15600]: 2007/08/08_08:54:18 info: Exiting status process 21247 returned rc 0.
heartbeat[15600]: 2007/08/08_08:54:18 debug: RscMgmtProc 'status' exited code 0
heartbeat[15600]: 2007/08/08_08:54:18 debug: StartNextRemoteRscReq() - calling hook
heartbeat[15600]: 2007/08/08_08:54:18 debug: notify_world: invoking harc: OLD status: active
heartbeat[15600]: 2007/08/08_08:54:18 debug: Process [status] started pid 21262
heartbeat[15600]: 2007/08/08_08:54:18 debug: Starting notify process [status]
heartbeat[21262]: 2007/08/08_08:54:18 debug: notify_world: setting SIGCHLD Handler to SIG_DFL
heartbeat[21262]: 2007/08/08_08:54:18 debug: notify_world: Running harc status
harc[21262]:	2007/08/08_08:54:18 info: Running /usr/local/etc/ha.d/rc.d/status status
heartbeat[15600]: 2007/08/08_08:54:18 info: Exiting status process 21262 returned rc 0.
heartbeat[15600]: 2007/08/08_08:54:18 debug: RscMgmtProc 'status' exited code 0
heartbeat[15600]: 2007/08/08_08:54:18 info: Link linux6:/dev/ttyS0 up.
heartbeat[15600]: 2007/08/08_08:54:19 debug: Get a reqnodes message from linux6
heartbeat[15600]: 2007/08/08_08:54:19 debug: get_delnodelist: delnodelist= 
heartbeat[15600]: 2007/08/08_08:54:20 info: Status update for node linux6: status active
heartbeat[15600]: 2007/08/08_08:54:20 debug: Status seqno: 6 msgtime: 1186534459
heartbeat[15600]: 2007/08/08_08:54:20 debug: StartNextRemoteRscReq() - calling hook
heartbeat[15600]: 2007/08/08_08:54:20 debug: notify_world: invoking harc: OLD status: active
heartbeat[15600]: 2007/08/08_08:54:20 debug: Process [status] started pid 21277
heartbeat[15600]: 2007/08/08_08:54:20 debug: Starting notify process [status]
heartbeat[15600]: 2007/08/08_08:54:20 debug: process_resources: other now unstable
heartbeat[15600]: 2007/08/08_08:54:20 WARN: T_STARTING received during takeover.
heartbeat[15600]: 2007/08/08_08:54:20 debug: Sending hold resources msg: all, stable=1 # <none>
heartbeat[15600]: 2007/08/08_08:54:20 info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1))
heartbeat[15600]: 2007/08/08_08:54:20 debug: hb_rsc_isstable: ResourceMgmt_child_count: 1, other_is_stable: 0, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:20 debug: hb_rsc_isstable: ResourceMgmt_child_count: 1, other_is_stable: 0, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[21277]: 2007/08/08_08:54:20 debug: notify_world: setting SIGCHLD Handler to SIG_DFL
heartbeat[21277]: 2007/08/08_08:54:20 debug: notify_world: Running harc status
harc[21277]:	2007/08/08_08:54:20 info: Running /usr/local/etc/ha.d/rc.d/status status
heartbeat[15600]: 2007/08/08_08:54:20 info: Exiting status process 21277 returned rc 0.
heartbeat[15600]: 2007/08/08_08:54:20 debug: RscMgmtProc 'status' exited code 0
heartbeat[15600]: 2007/08/08_08:54:20 debug: process_resources(2):  other now unstable
heartbeat[15600]: 2007/08/08_08:54:20 info: other_holds_resources: 0
heartbeat[15600]: 2007/08/08_08:54:20 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 0, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:20 info: remote resource transition completed.
heartbeat[15600]: 2007/08/08_08:54:20 debug: Sending hold resources msg: all, stable=1 # <none>
heartbeat[15600]: 2007/08/08_08:54:20 info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1))
heartbeat[15600]: 2007/08/08_08:54:20 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:20 debug: Calling PerformAutoFailback()
heartbeat[15600]: 2007/08/08_08:54:20 info: other_holds_resources: 0
heartbeat[15600]: 2007/08/08_08:54:20 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(msgfmt,netstring)
heartbeat[21427]: 2007/08/08_08:54:24 debug: uid=hacluster, gid=<null>
heartbeat[21427]: 2007/08/08_08:54:24 debug: uid=<null>, gid=haclient
heartbeat[21427]: 2007/08/08_08:54:24 debug: uid=root, gid=<null>
heartbeat[21427]: 2007/08/08_08:54:24 debug: uid=<null>, gid=haclient
heartbeat[21427]: 2007/08/08_08:54:24 debug: Beginning authentication parsing
heartbeat[21427]: 2007/08/08_08:54:24 debug: 16 max authentication methods
heartbeat[21427]: 2007/08/08_08:54:24 debug: Keyfile opened
heartbeat[21427]: 2007/08/08_08:54:24 debug: Keyfile perms OK
heartbeat[21427]: 2007/08/08_08:54:24 debug: 16 max authentication methods
heartbeat[21427]: 2007/08/08_08:54:24 debug: Found authentication method [crc]
heartbeat[21427]: 2007/08/08_08:54:24 info: AUTH: i=1: key = 0x5d7cc8, auth=0x2a9619c5c0, authname=crc
heartbeat[21427]: 2007/08/08_08:54:24 debug: Outbound signing method is 1
heartbeat[21427]: 2007/08/08_08:54:24 debug: Authentication parsing complete [1]
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(cluster,linux-ha)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(hopfudge,1)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(hbgenmethod,file)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(realtime,true)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(conn_logd_time,60)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(log_badpack,true)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(coredumps,true)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(crm,false)
heartbeat[21427]: 2007/08/08_08:54:24 info: Version 2 support: false
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(autojoin,none)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(uuidfrom,file)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(compression,zlib)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(compression_threshold,2)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(traditional_compression,no)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(max_rexmit_delay,250)
heartbeat[21427]: 2007/08/08_08:54:24 debug: Setting max_rexmit_delay to 250 ms
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(record_config_changes,on)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(record_pengine_inputs,on)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(enable_config_writes,on)
heartbeat[21427]: 2007/08/08_08:54:24 debug: add_option(memreserve,6500)
heartbeat[21427]: 2007/08/08_08:54:24 WARN: Logging daemon is disabled --enabling logging daemon is recommended
heartbeat[21427]: 2007/08/08_08:54:24 info: **************************
heartbeat[21427]: 2007/08/08_08:54:24 info: Configuration validated. Starting heartbeat 2.1.2
heartbeat[21427]: 2007/08/08_08:54:24 debug: HA configuration OK.  Heartbeat starting.
heartbeat[21427]: 2007/08/08_08:54:24 info: heartbeat: already running [pid 15600].
heartbeat[15600]: 2007/08/08_08:54:44 debug: Received standby message me from linux7 in state 0 
heartbeat[15600]: 2007/08/08_08:54:44 debug: ask_for_resources: other now unstable
heartbeat[15600]: 2007/08/08_08:54:44 info: linux7 wants to go standby [all]
heartbeat[15600]: 2007/08/08_08:54:44 info: i_hold_resources: 3
heartbeat[15600]: 2007/08/08_08:54:44 info: New standby state: 1
heartbeat[15600]: 2007/08/08_08:54:45 debug: Received standby message other from linux6 in state 1 
heartbeat[15600]: 2007/08/08_08:54:45 info: standby: linux6 can take our all resources
heartbeat[15600]: 2007/08/08_08:54:45 debug: go_standby: other is unstable
heartbeat[15600]: 2007/08/08_08:54:45 debug: Sending hold resources msg: none, stable=0 # standby
heartbeat[15600]: 2007/08/08_08:54:45 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 0, takeover_in_progress: 1, going_standby: 1, standby running(ms): 4445781550, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:45 debug: Process [go_standby] started pid 21442
heartbeat[15600]: 2007/08/08_08:54:45 info: New standby state: 1
heartbeat[21442]: 2007/08/08_08:54:45 info: give up all HA resources (standby).
heartbeat[21442]: 2007/08/08_08:54:45 info: go_standby: who: 1 resource set: all
heartbeat[21442]: 2007/08/08_08:54:45 info: go_standby: (query/action): (allkeys/givegroup)
ResourceManager[21455]:	2007/08/08_08:54:45 info: Releasing resource group: linux6 172.21.50.188/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data0::ext3 ChangeSrcIP::172.21.50.58::32::eth0::0 ChangeSrcIP::172.21.60.190::32::eth0::0 drbddisk::r1 Filesystem::/dev/drbd1::/data1::ext3 cdrecvd cdsendd
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/cdsendd  stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/cdsendd  stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/cdsendd  stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/cdrecvd  stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/cdrecvd  stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/cdrecvd  stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 stop
Filesystem[21545]:	2007/08/08_08:54:45 INFO: Running stop for /dev/drbd1 on /data1
Filesystem[21534]:	2007/08/08_08:54:45 INFO:  Success
INFO:  Success
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/drbddisk r1 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/drbddisk r1 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/drbddisk r1 stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 stop
ChangeSrcIP[21645]:	2007/08/08_08:54:45 INFO: Running stop for managing routes.
ChangeSrcIP[21634]:	2007/08/08_08:54:45 INFO:  Success
INFO:  Success
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 stop
ChangeSrcIP[21701]:	2007/08/08_08:54:45 INFO: Running stop for managing routes.
ChangeSrcIP[21690]:	2007/08/08_08:54:45 INFO:  Success
INFO:  Success
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 stop
Filesystem[21757]:	2007/08/08_08:54:45 INFO: Running stop for /dev/drbd0 on /data0
Filesystem[21757]:	2007/08/08_08:54:45 INFO: Trying to unmount /data0
Filesystem[21757]:	2007/08/08_08:54:45 INFO: unmounted /data0 successfully
Filesystem[21746]:	2007/08/08_08:54:45 INFO:  Success
INFO:  Success
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/drbddisk r0 stop done. RC=0
ResourceManager[21455]:	2007/08/08_08:54:45 info: Running /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 stop
ResourceManager[21455]:	2007/08/08_08:54:45 debug: Starting /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 stop
In IP Stop
SIOCDELRT: No such process
IPaddr[21892]:	2007/08/08_08:54:45 INFO: ifconfig eth0:0 down
IPaddr[21863]:	2007/08/08_08:54:45 INFO:  Success
INFO:  Success
ResourceManager[21455]:	2007/08/08_08:54:45 debug: /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 stop done. RC=0
heartbeat[21442]: 2007/08/08_08:54:45 info: all HA resource release completed (standby).
heartbeat[21442]: 2007/08/08_08:54:45 debug: Sending standby [done] msg
heartbeat[21442]: 2007/08/08_08:54:45 info: FIFO message [type ask_resources] written rc=64
heartbeat[15600]: 2007/08/08_08:54:45 debug: Received standby message done from linux7 in state 1 
heartbeat[15600]: 2007/08/08_08:54:45 info: Local standby process completed [all].
heartbeat[15600]: 2007/08/08_08:54:45 info: New standby state: 3
heartbeat[15600]: 2007/08/08_08:54:45 info: Exiting go_standby process 21442 returned rc 0.
heartbeat[15600]: 2007/08/08_08:54:45 debug: RscMgmtProc 'go_standby' exited code 0
heartbeat[15600]: 2007/08/08_08:54:47 WARN: 1 lost packet(s) for [linux6] [26:28]
heartbeat[15600]: 2007/08/08_08:54:47 info: remote resource transition completed.
heartbeat[15600]: 2007/08/08_08:54:47 debug: Sending hold resources msg: none, stable=1 # <none>
heartbeat[15600]: 2007/08/08_08:54:47 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 1, going_standby: 3, standby running(ms): 4445782140, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:47 debug: Calling PerformAutoFailback()
heartbeat[15600]: 2007/08/08_08:54:47 info: other_holds_resources: 3
heartbeat[15600]: 2007/08/08_08:54:47 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 1, going_standby: 3, standby running(ms): 4445782140, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:47 info: No pkts missing from linux6!
heartbeat[15600]: 2007/08/08_08:54:47 debug: Received standby message done from linux6 in state 3 
heartbeat[15600]: 2007/08/08_08:54:47 info: Other node completed standby takeover of all resources.
heartbeat[15600]: 2007/08/08_08:54:47 debug: Sending hold resources msg: none, stable=1 # <none>
heartbeat[15600]: 2007/08/08_08:54:47 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:54:47 info: New standby state: 0
heartbeat[15600]: 2007/08/08_08:54:48 info: other_holds_resources: 3
heartbeat[15600]: 2007/08/08_08:54:48 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: ha_msg_addraw_ll(): no closing parentheses
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: ha_msg_addraw_ll: addfield failed
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: ha_msg_addraw(): ha_msg_addraw_ll failed
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: NV failure (string2msg_ll):
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: Input string: [>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoD5t1nRSmJ2wPAn5cTwA=24
hg=46b83d1e
ts=46b91465
ld=0.21 0.05 0.02 1/105 7743
ttl=4
auth=13<>>>
t=NS_ackmsg
sinux7
ackseq=555f
(1el9zwP6T3OlazZLKnCIr61)srcuuid=uoDQ5t1nRSmJ2wPAncg1
ts=46b91467
ttl=4
auth=1 f2c<>>>
t=status
st=c=r1ns=nwPAn5cTwA==
seq=25
hg=46b83d1e
ts=46b91467
ld=0. 0 th21ee
<<<
]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: sp=(1el9zwP6T3OlazZLKnCIr61)srcuuid=uoDQ5t1nRSmJ2wPAncg1
ts=46b91467
ttl=4
auth=1 f2c<>>>
t=status
st=c=r1ns=nwPAn5cTwA==
seq=25
hg=46b83d1e
ts=46b91467
ld=0. 0 th21ee
<<<

heartbeat[15600]: 2007/08/08_08:55:03 ERROR: depth=0
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG: Dumping message with 14 fields
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[1] : [st=active]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[2] : [dt=2710]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[3] : [protocol=1]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[4] : [src=linux6]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[5] : [(1)srcuuid=0x60d5f8(12 10)]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[6] : [hg=46b83d1e]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[7] : [ts=46b91465]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[8] : [ld=0.21 0.05 0.02 1/105 7743]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[9] : [ttl=4]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[10] : [auth=13<>>>]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[11] : [t=NS_ackmsg]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[12] : [sinux7
ackseq=555f]
heartbeat[15600]: 2007/08/08_08:55:03 ERROR: MSG[13] : [ackseq=555f]
heartbeat[15600]: 2007/08/08_08:55:05 WARN: string2msg_ll: node [?] failed authentication
heartbeat[15600]: 2007/08/08_08:55:07 WARN: Bad/invalid auth token, authtoken=(nil)
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG: Dumping message with 6 fields
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG[1] : [st=1cri)=nPwA==]
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG[2] : [seq=27]
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG[3] : [hg=46b83d1e]
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG[4] : [ts116/4
h=1 3fe51fd3]
heartbeat[15600]: 2007/08/08_08:55:07 info: MSG[5] : [h=1 3fe51fd3]
heartbeat[15600]: 2007/08/08_08:55:07 WARN: string2msg_ll: node [?] failed authentication
heartbeat[15600]: 2007/08/08_08:55:11 WARN: Bad/invalid auth token, authtoken=(nil)
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG: Dumping message with 14 fields
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[1] : [st=active]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[2] : [dt=2710]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[3] : [protocol=1]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[4] : [c
u5J
3=d=1.17 0.26 0.08 2/105 7749]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[5] : [u5J
3=d=1.17 0.26 0.08 2/105 7749]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[6] : [3=d=1.17 0.26 0.08 2/105 7749]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[7] : [ttl=4]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[8] : [at1<>>>
t=status]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[9] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[10] : [st=e0r1nsut1nRSmJ2wPAn5cTwA==]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[11] : [seq=29]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[12] : [g1bd0 .08 1/105 7749
ttl=4h=1 13660875]
heartbeat[15600]: 2007/08/08_08:55:11 info: MSG[13] : [ttl=4h=1 13660875]
heartbeat[15600]: 2007/08/08_08:55:11 WARN: string2msg_ll: node [?] failed authentication
heartbeat[15600]: 2007/08/08_08:55:16 WARN: ha_msg_add_nv_depth: line doesn't contain '='
heartbeat[15600]: 2007/08/08_08:55:16 info: =linux6
(1)srcuuid=uQmn5cTA=
q=2b
hg=46b83d1e
ts=6b91473
ld=1.48 0.34 0.11 1/105 7751
ttl=4
auth=1 e22e61bc
<<<

heartbeat[15600]: 2007/08/08_08:55:16 ERROR: NV failure (string2msg_ll):
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: Input string: [>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=2a
hg=46b83d1e
ts=46b91471
ld=1.48 0.34 0.11 1/105 7751
ttl=4
auth=1 61a6e0f2
<<>>>
t=status
st=c=o
=linux6
(1)srcuuid=uQmn5cTA=
q=2b
hg=46b83d1e
ts=6b91473
ld=1.48 0.34 0.11 1/105 7751
ttl=4
auth=1 e22e61bc
<<<
]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: sp==linux6
(1)srcuuid=uQmn5cTA=
q=2b
hg=46b83d1e
ts=6b91473
ld=1.48 0.34 0.11 1/105 7751
ttl=4
auth=1 e22e61bc
<<<

heartbeat[15600]: 2007/08/08_08:55:16 ERROR: depth=0
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG: Dumping message with 15 fields
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[1] : [st=active]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[2] : [dt=2710]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[3] : [protocol=1]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[4] : [src=linux6]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[5] : [(1)srcuuid=0x609d38(36 27)]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[6] : [seq=2a]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[7] : [hg=46b83d1e]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[8] : [ts=46b91471]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[9] : [ld=1.48 0.34 0.11 1/105 7751]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[10] : [ttl=4]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[11] : [auth=1 61a6e0f2]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[12] : [<<>>>
t=status]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[13] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:16 ERROR: MSG[14] : [st=c=o]
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: ha_msg_addraw_ll(): no closing parentheses
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: ha_msg_addraw_ll: addfield failed
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: ha_msg_addraw(): ha_msg_addraw_ll failed
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: NV failure (string2msg_ll):
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: Input string: [>>>
t=status
st=ae1rotocol=1
src=linux6
()=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=2c
hg=46b83d1e
ts=46b9147l01 l4
auth=1 2c453141
<<>>>
t=NS_ackmsg
dest=linux6eq=2d
(1)destuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
src(i=uoDQ5t1nRSmJ2wPAn5cTwA==
hg=46b83d6b91477
ttl=4
auth=1 4f1a89a4
<<<
]
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: sp=()=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=2c
hg=46b83d1e
ts=46b9147l01 l4
auth=1 2c453141
<<>>>
t=NS_ackmsg
dest=linux6eq=2d
(1)destuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
src(i=uoDQ5t1nRSmJ2wPAn5cTwA==
hg=46b83d6b91477
ttl=4
auth=1 4f1a89a4
<<<

heartbeat[15600]: 2007/08/08_08:55:20 ERROR: depth=0
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: MSG: Dumping message with 3 fields
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: MSG[1] : [st=ae1rotocol=1]
heartbeat[15600]: 2007/08/08_08:55:20 ERROR: MSG[2] : [src=linux6]
heartbeat[15600]: 2007/08/08_08:55:22 WARN: Bad/invalid auth token, authtoken=(nil)
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG: Dumping message with 19 fields
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[1] : [st=dpll]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[2] : [(1)srcuuid=0x60fcf8(36 27)]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[3] : [seq=2d]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[4] : [hg=46b8d=46b91477]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[5] : [ld=1.44 0.35 0.11 1/105 7753]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[6] : [ttl=4]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[7] : [u9<>>>
t=status]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[8] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[9] : [st=active]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[10] : [dt=2710]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[11] : [protocol=1]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[12] : [src=ux6]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[13] : [(1)srcuuid=0x611bc8(12 10)]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[14] : [seq=2e]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[15] : [hg=46b83d1e]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[16] : [ts=46b91479]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[17] : [ld=1.44 0.35 0.11 2/105 7753]
heartbeat[15600]: 2007/08/08_08:55:22 info: MSG[18] : [lth=1 e245b5e2]
heartbeat[15600]: 2007/08/08_08:55:22 WARN: string2msg_ll: node [ux6] failed authentication
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: ha_msg_addraw_ll: illegal field
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: ha_msg_addraw(): ha_msg_addraw_ll failed
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: NV failure (string2msg_ll):
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: Input string: [>>>
t=status
st=c=o1nsuRA=
34b0.5tt5<=slc5elTlIc
uQJTg1b7
 2<
>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=30
hg=46b83d1e
ts=46b9147d
ld=1.40 0.36 0.12 1/105 7753
ttl=4
auth=1 76f5a231
<<<
]
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: sp=>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=30
hg=46b83d1e
ts=46b9147d
ld=1.40 0.36 0.12 1/105 7753
ttl=4
auth=1 76f5a231
<<<

heartbeat[15600]: 2007/08/08_08:55:25 ERROR: depth=0
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: MSG: Dumping message with 5 fields
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: MSG[1] : [st=c=o1nsuRA=]
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: MSG[2] : [34b0.5tt5<=slc5elTlIc]
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: MSG[3] : [uQJTg1b7
 2<
>>>
t=status]
heartbeat[15600]: 2007/08/08_08:55:25 ERROR: MSG[4] : [ 2<
>>>
t=status]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: ha_msg_addraw_ll: illegal field
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: ha_msg_addraw(): ha_msg_addraw_ll failed
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: NV failure (string2msg_ll):
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: Input string: [>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=dc
hg=46b83d1e
ts=46b915d5
ld=0.03 0.14 0.08 1/105 7772
ttl=a9<
>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=dd
hg=46b83d1e
ts=46b915d7
ld=0.03 0.14 0.08 1/105 7772
ttl=4
auth=1 2f1f5f90
<<<
]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: sp=>>>
t=status
st=active
dt=2710
protocol=1
src=linux6
(1)srcuuid=uoDQ5t1nRSmJ2wPAn5cTwA==
seq=dd
hg=46b83d1e
ts=46b915d7
ld=0.03 0.14 0.08 1/105 7772
ttl=4
auth=1 2f1f5f90
<<<

heartbeat[15600]: 2007/08/08_09:01:12 ERROR: depth=0
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG: Dumping message with 11 fields
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[0] : [t=status]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[1] : [st=active]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[2] : [dt=2710]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[3] : [protocol=1]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[4] : [src=linux6]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[5] : [(1)srcuuid=0x60ed98(36 27)]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[6] : [seq=dc]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[7] : [hg=46b83d1e]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[8] : [ts=46b915d5]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[9] : [ld=0.03 0.14 0.08 1/105 7772]
heartbeat[15600]: 2007/08/08_09:01:12 ERROR: MSG[10] : [ttl=a9<]
heartbeat[15600]: 2007/08/08_09:02:01 info: Daily informational memory statistics
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 12/178180 ms age 0 [pid15600/MST_CONTROL]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 850/6350163  174776/92566 [pid15600/MST_CONTROL]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 466080 total malloc bytes. pid [15600/MST_CONTROL]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/9 ms age 435620 [pid15603/HBFIFO]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 458/679  63800/34927 [pid15603/HBFIFO]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 66352 total malloc bytes. pid [15603/HBFIFO]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/26407 ms age 330 [pid15604/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 441/1111540  64124/35926 [pid15604/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 76004 total malloc bytes. pid [15604/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/0 ms age 4442617770 [pid15605/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 442/51603  64216/35990 [pid15605/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 66492 total malloc bytes. pid [15605/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/0 ms age 4442617770 [pid15606/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 443/28474  64308/36054 [pid15606/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 112984 total malloc bytes. pid [15606/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/0 ms age 4442617770 [pid15607/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 444/104657  64400/36118 [pid15607/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 65532 total malloc bytes. pid [15607/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/0 ms age 4442617770 [pid15608/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 445/28483  64492/36182 [pid15608/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 113168 total malloc bytes. pid [15608/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/0 ms age 4442617770 [pid15609/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 446/104667  64584/36246 [pid15609/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 65716 total malloc bytes. pid [15609/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 1/48446 ms age 0 [pid15610/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 477/1473091  67756/37763 [pid15610/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 78264 total malloc bytes. pid [15610/HBWRITE]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: MSG stats: 0/44074 ms age 330 [pid15611/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: cl_malloc stats: 448/1102377  64768/36374 [pid15611/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: RealMalloc stats: 66932 total malloc bytes. pid [15611/HBREAD]
heartbeat[15600]: 2007/08/08_09:02:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:02:01 info: These are nothing to worry about.
heartbeat[15600]: 2007/08/08_09:02:55 WARN: node linux6: is dead
heartbeat[15600]: 2007/08/08_09:02:55 WARN: No STONITH device configured.
heartbeat[15600]: 2007/08/08_09:02:55 WARN: Shared disks are not protected.
heartbeat[15600]: 2007/08/08_09:02:55 info: Resources being acquired from linux6.
heartbeat[15600]: 2007/08/08_09:02:55 debug: StartNextRemoteRscReq() - calling hook
heartbeat[15600]: 2007/08/08_09:02:55 debug: notify_world: invoking harc: OLD status: active
heartbeat[15600]: 2007/08/08_09:02:55 debug: Process [status] started pid 21957
heartbeat[15600]: 2007/08/08_09:02:55 debug: Starting notify process [status]
heartbeat[15600]: 2007/08/08_09:02:55 debug: takeover_from_node: other now stable
heartbeat[15600]: 2007/08/08_09:02:55 debug: Process [req_our_resources] started pid 21958
heartbeat[21957]: 2007/08/08_09:02:55 debug: notify_world: setting SIGCHLD Handler to SIG_DFL
heartbeat[21957]: 2007/08/08_09:02:55 debug: notify_world: Running harc status
heartbeat[21958]: 2007/08/08_09:02:55 debug: req_our_resources(/usr/local/share/heartbeat/ResourceManager listkeys linux7)
harc[21957]:	2007/08/08_09:02:55 info: Running /usr/local/etc/ha.d/rc.d/status status
heartbeat[21958]: 2007/08/08_09:02:55 info: No local resources [/usr/local/share/heartbeat/ResourceManager listkeys linux7] to acquire.
heartbeat[21958]: 2007/08/08_09:02:55 debug: Sending hold resources msg: all, stable=1 # req_our_resources()
heartbeat[21958]: 2007/08/08_09:02:55 info: FIFO message [type resource] written rc=108
heartbeat[15600]: 2007/08/08_09:02:55 info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1))
heartbeat[15600]: 2007/08/08_09:02:55 debug: hb_rsc_isstable: ResourceMgmt_child_count: 2, other_is_stable: 1, takeover_in_progress: 1, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_09:02:55 info: Exiting req_our_resources process 21958 returned rc 0.
heartbeat[15600]: 2007/08/08_09:02:55 debug: RscMgmtProc 'req_our_resources' exited code 0
heartbeat[15600]: 2007/08/08_09:02:55 info: AnnounceTakeover(local 1, foreign 1, reason 'req_our_resources' (1))
heartbeat[15600]: 2007/08/08_09:02:55 debug: StartNextRemoteRscReq(): child count 1
mach_down[21979]:	2007/08/08_09:02:55 info: Taking over resource group 172.21.50.188/24/eth0
ResourceManager[22010]:	2007/08/08_09:02:55 info: Acquiring resource group: linux6 172.21.50.188/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data0::ext3 ChangeSrcIP::172.21.50.58::32::eth0::0 ChangeSrcIP::172.21.60.190::32::eth0::0 drbddisk::r1 Filesystem::/dev/drbd1::/data1::ext3 cdrecvd cdsendd
IPaddr[22036]:	2007/08/08_09:02:55 INFO:  Resource is stopped
ResourceManager[22010]:	2007/08/08_09:02:55 info: Running /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 start
ResourceManager[22010]:	2007/08/08_09:02:55 debug: Starting /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 start
IPaddr[22133]:	2007/08/08_09:02:56 INFO: Using calculated netmask for 172.21.50.188: 255.255.255.0
IPaddr[22133]:	2007/08/08_09:02:56 DEBUG: Using calculated broadcast for 172.21.50.188: 172.21.50.255
IPaddr[22133]:	2007/08/08_09:02:56 INFO: eval ifconfig eth0:0 172.21.50.188 netmask 255.255.255.0 broadcast 172.21.50.255
IPaddr[22133]:	2007/08/08_09:02:56 DEBUG: Sending Gratuitous Arp for 172.21.50.188 on eth0:0 [eth0]
IPaddr[22104]:	2007/08/08_09:02:56 INFO:  Success
INFO:  Success
ResourceManager[22010]:	2007/08/08_09:02:56 debug: /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 start done. RC=0
ResourceManager[22010]:	2007/08/08_09:02:56 info: Running /usr/local/etc/ha.d/resource.d/drbddisk r0 start
ResourceManager[22010]:	2007/08/08_09:02:56 debug: Starting /usr/local/etc/ha.d/resource.d/drbddisk r0 start
ResourceManager[22010]:	2007/08/08_09:02:56 debug: /usr/local/etc/ha.d/resource.d/drbddisk r0 start done. RC=0
Filesystem[22271]:	2007/08/08_09:02:56 INFO:  Resource is stopped
ResourceManager[22010]:	2007/08/08_09:02:56 info: Running /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 start
ResourceManager[22010]:	2007/08/08_09:02:56 debug: Starting /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 start
Filesystem[22348]:	2007/08/08_09:02:56 INFO: Running start for /dev/drbd0 on /data0
Filesystem[22337]:	2007/08/08_09:02:57 INFO:  Success
INFO:  Success
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 start done. RC=0
ChangeSrcIP[22414]:	2007/08/08_09:02:57 INFO:  Resource is stopped
ResourceManager[22010]:	2007/08/08_09:02:57 info: Running /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: Starting /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 start
ChangeSrcIP[22475]:	2007/08/08_09:02:57 INFO: Running start for managing routes.
ChangeSrcIP[22464]:	2007/08/08_09:02:57 INFO:  Success
INFO:  Success
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 start done. RC=0
ChangeSrcIP[22513]:	2007/08/08_09:02:57 INFO:  Resource is stopped
ResourceManager[22010]:	2007/08/08_09:02:57 info: Running /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: Starting /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 start
ChangeSrcIP[22574]:	2007/08/08_09:02:57 INFO: Running start for managing routes.
ChangeSrcIP[22563]:	2007/08/08_09:02:57 INFO:  Success
INFO:  Success
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 start done. RC=0
ResourceManager[22010]:	2007/08/08_09:02:57 info: Running /usr/local/etc/ha.d/resource.d/drbddisk r1 start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: Starting /usr/local/etc/ha.d/resource.d/drbddisk r1 start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/drbddisk r1 start done. RC=0
Filesystem[22650]:	2007/08/08_09:02:57 INFO:  Resource is stopped
ResourceManager[22010]:	2007/08/08_09:02:57 info: Running /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: Starting /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 start
Filesystem[22727]:	2007/08/08_09:02:57 INFO: Running start for /dev/drbd1 on /data1
Filesystem[22716]:	2007/08/08_09:02:57 INFO:  Success
INFO:  Success
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 start done. RC=0
ResourceManager[22010]:	2007/08/08_09:02:57 info: Running /usr/local/etc/ha.d/resource.d/cdrecvd  start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: Starting /usr/local/etc/ha.d/resource.d/cdrecvd  start
(Debug:01)(182894245248) CDreceiver 2.0.4 started at 08/08/07 01:02 by User 500
(Debug:01)(1084229984) Monitor thread started
(Debug:01)(1094719840) Listener started, listen at 0.0.0.0:8000
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/cdrecvd  start done. RC=0
ResourceManager[22010]:	2007/08/08_09:02:57 info: Running /usr/local/etc/ha.d/resource.d/cdsendd  start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: Starting /usr/local/etc/ha.d/resource.d/cdsendd  start
ResourceManager[22010]:	2007/08/08_09:02:57 debug: /usr/local/etc/ha.d/resource.d/cdsendd  start done. RC=0
mach_down[21979]:	2007/08/08_09:02:57 info: /usr/local/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[21979]:	2007/08/08_09:02:57 info: mach_down takeover complete for node linux6.
heartbeat[15600]: 2007/08/08_09:02:57 info: AnnounceTakeover(local 1, foreign 1, reason 'T_RESOURCES(us)' (1))
heartbeat[15600]: 2007/08/08_09:02:57 info: mach_down takeover complete.
heartbeat[15600]: 2007/08/08_09:02:57 debug: process_resources(3):  other now stable
heartbeat[15600]: 2007/08/08_09:02:57 info: AnnounceTakeover(local 1, foreign 1, reason 'mach_down' (1))
heartbeat[15600]: 2007/08/08_09:02:57 debug: hb_rsc_isstable: ResourceMgmt_child_count: 1, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_09:02:57 info: Exiting status process 21957 returned rc 0.
heartbeat[15600]: 2007/08/08_09:02:57 debug: RscMgmtProc 'status' exited code 0
(Debug:01)(182895382912) CDsender 2.0.4 started at 08/08/07 01:02 by User 500
(Debug:01)(1094719840) Command thread started, listen at 0.0.0.0:8001
(Debug:01)(1084229984) Scheduler started
(Debug:01)(1105209696) [HILR_Linux4] mode 0, last clf file parsed [/data0/08/HILR.20070808.0054] and offset 25404288
(Debug:01)(1105209696) [HILR_Linux4] last frame sent filename [/data0/08/HILR.20070808.0054] and offset 24544288; RNT 1186533070.000000
(Debug:01)(1105209696) [HILR_Linux4] restart file found in clflist: [/data0/08/HILR.20070808.0054]
(Debug:01)(1136679264) [MKAR_Solaris] mode 0, last clf file parsed [/data0/08/MKAR.20070808.0054] and offset 63777536
(Debug:01)(1136679264) [MKAR_Solaris] last frame sent filename [/data0/08/MKAR.20070808.0054] and offset 63755892; RNT 1186534600.000000
(Debug:01)(1136679264) [MKAR_Solaris] restart file found in clflist: [/data0/08/MKAR.20070808.0054]
(Debug:01)(1115699552) [HILR_Solaris] mode 0, last clf file parsed [/data0/08/HILR.20070808.0054] and offset 25404288
(Debug:01)(1115699552) [HILR_Solaris] last frame sent filename [/data0/08/HILR.20070808.0054] and offset 24807704; RNT 1186533500.000000
(Debug:01)(1115699552) [HILR_Solaris] restart file found in clflist: [/data0/08/HILR.20070808.0054]
(Debug:01)(1126189408) [MKAR_Linux4] mode 0, last clf file parsed [/data0/08/MKAR.20070808.0054] and offset 63925908
(Debug:01)(1126189408) [MKAR_Linux4] last frame sent filename [/data0/08/MKAR.20070808.0054] and offset 63905684; RNT 1186534700.000000
(Debug:01)(1126189408) [MKAR_Linux4] restart file found in clflist: [/data0/08/MKAR.20070808.0054]
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38475 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38476 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38477 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38478 CD1.1 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38481 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38482 CD1.0 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38483 CD1.1 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38484 CD1.1 0 connection closed unexpected
(Debug:01)(1105209696) CON_I: HILR 172.21.50.57:40994 to 172.21.50.188:8002 CD1.0 1186534995 station connected
heartbeat[15600]: 2007/08/08_09:03:15 info: Link linux6:/dev/ttyS0 dead.
heartbeat[15600]: 2007/08/08_09:03:15 info: Link linux6:eth1 dead.
heartbeat[15600]: 2007/08/08_09:03:15 info: Link linux6:eth2 dead.
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38488 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38487 CD1.0 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38489 CD1.1 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38490 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CON_I: MKAR 172.21.50.57:40997 to 172.21.50.188:8003 CD1.1 1186534999 station connected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38495 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38496 CD1.0 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38497 CD1.1 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38498 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38501 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38502 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38504 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38503 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38509 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38510 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38511 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38512 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38515 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38516 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38517 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38518 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38525 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38526 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38527 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38528 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38529 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38530 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38531 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38532 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38539 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38540 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38541 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38542 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38543 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38544 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38545 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38546 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38553 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38554 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38555 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38556 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38558 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38559 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38560 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38561 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38568 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38569 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_E: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38570 CD1.1 0 connection closed unexpected
(Debug:01)(1136679264) CRQ_E: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38571 CD1.1 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38573 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38574 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CRQ_W: MKAR_Linux4 172.21.50.58:8000 to 172.21.50.188:38575 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_I: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 station connected
(Debug:01)(1136679264) CRQ_W: MKAR_Solaris 172.21.60.190:10000 to 172.21.50.188:38577 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_I: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 station connected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38585 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38586 CD1.0 0 connection closed unexpected
heartbeat[15604]: 2007/08/08_09:05:35 WARN: glib: TTY write timeout on [/dev/ttyS0] (no connection or bad cable? [see documentation])
heartbeat[15604]: 2007/08/08_09:05:35 info: glib: See http://linux-ha.org/FAQ#TTYtimeout for details
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38589 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38590 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38595 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38596 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38599 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38600 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38605 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38606 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38610 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38611 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38616 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38617 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38620 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38623 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38626 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38627 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38632 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38633 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38636 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38637 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38642 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38643 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38646 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38647 CD1.0 0 connection closed unexpected
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38652 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38653 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38656 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38657 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38664 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38665 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38666 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38667 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38674 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38675 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38676 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38677 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38684 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38685 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38686 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38687 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38694 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38695 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38696 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38697 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38704 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38705 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38709 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38710 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38715 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38716 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38719 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38720 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38725 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38726 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38729 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38730 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38735 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38736 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38741 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38742 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38745 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38746 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38751 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38752 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38755 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38756 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38761 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38762 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38765 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38766 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38771 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38772 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38775 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38776 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38783 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38784 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38785 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38786 CD1.0 0 connection closed unexpected
heartbeat[15600]: 2007/08/08_09:12:01 info: Daily informational memory statistics
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 100/179384 ms age 0 [pid15600/MST_CONTROL]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 3314/6390717  412728/204414 [pid15600/MST_CONTROL]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 466080 total malloc bytes. pid [15600/MST_CONTROL]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/11 ms age 543830 [pid15603/HBFIFO]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 463/723  64100/35018 [pid15603/HBFIFO]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 66832 total malloc bytes. pid [15603/HBFIFO]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/26747 ms age 0 [pid15604/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 441/1125977  64124/35926 [pid15604/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 76004 total malloc bytes. pid [15604/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/0 ms age 4443217760 [pid15605/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 442/51659  64216/35990 [pid15605/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 66492 total malloc bytes. pid [15605/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/0 ms age 4443217760 [pid15606/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 445/28727  64492/36150 [pid15606/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 112984 total malloc bytes. pid [15606/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/0 ms age 4443217760 [pid15607/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 444/105185  64400/36118 [pid15607/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 65532 total malloc bytes. pid [15607/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/0 ms age 4443217760 [pid15608/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 447/28735  64676/36278 [pid15608/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 113168 total malloc bytes. pid [15608/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/0 ms age 4443217760 [pid15609/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 446/105193  64584/36246 [pid15609/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 65716 total malloc bytes. pid [15609/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/49085 ms age 70 [pid15610/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 447/1492444  64676/36310 [pid15610/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 78264 total malloc bytes. pid [15610/HBWRITE]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: MSG stats: 0/44678 ms age 70 [pid15611/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: cl_malloc stats: 448/1117477  64768/36374 [pid15611/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: RealMalloc stats: 66932 total malloc bytes. pid [15611/HBREAD]
heartbeat[15600]: 2007/08/08_09:12:01 info: Current arena value: 0
heartbeat[15600]: 2007/08/08_09:12:01 info: These are nothing to worry about.
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38793 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38794 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38795 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38796 CD1.0 0 connection closed unexpected
(Debug:01)(1115699552) CRQ_E: HILR_Solaris 172.21.60.190:10000 to 172.21.50.188:38804 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38805 CD1.0 0 connection closed unexpected
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1115699552) CON_I: HILR_Solaris 172.21.60.190:10003 to 172.21.50.188:38807 CD1.0 0 station connected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38808 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38815 CD1.0 0 connection closed unexpected
(Debug:01)(1084229984) HILR: Connection info: connected, CD1.0, 8002, unknown
(Debug:01)(1084229984) MKAR: Connection info: connected, CD1.1, 8003, unknown
(Debug:01)(1084229984) Summary: Receiver started: 08/08/07 01:02
(Debug:01)(1084229984) Summary: -------- Listener values --------
(Debug:01)(1084229984) Summary: Listen on: 0.0.0.0:8000
(Debug:01)(1084229984) _U_8000: Connection requests warnings: 0
(Debug:01)(1084229984) _U_8000: Connection requestss failed: 0 [0/0/0/0/0/0/0/0]
(Debug:01)(1084229984) Summary: Received 0 frames [0/0/0/0/0/0/0/0/0/0/0], 0 bytes
(Debug:01)(1084229984) Summary: Sent 0 frames [0/0/0/0/0/0/0/0/0/0/0], 0 bytes
(Debug:01)(1084229984) Summary: -------- CD1 values --------
(Debug:01)(1084229984) Summary: Stations: 1 [0/0/0/1]
(Debug:01)(1084229984) Summary: Connection requests valid: 1
(Debug:01)(1084229984) Summary: Connection request warnings: 0
(Debug:01)(1084229984) Summary: Connection requests failed: 0 [0/0/0/0/0/0/0/0]
(Debug:01)(1084229984) Summary: Connections valid: 0
(Debug:01)(1084229984) Summary: Connection warnings: 0
(Debug:01)(1084229984) Summary: Connections failed: 0 [0/0/0/0/0/0/0/0]
(Debug:01)(1084229984) Summary: Received 70 frames [1/0/1/68/0/0/0/0/0/0/0], 440848 bytes
(Debug:01)(1084229984) Summary: Input buffer size: 10240 bytes
(Debug:01)(1084229984) Summary: Sent 1 frames [0/1/0/0/0/0/0/0/0/0/0], 32 bytes
(Debug:01)(1084229984) Summary: Cached Keys: 0
(Debug:01)(1084229984) Summary: ------- CD11 values --------
(Debug:01)(1084229984) Summary: Stations: 1 [0/0/0/1]
(Debug:01)(1084229984) Summary: Connection requests valid: 1
(Debug:01)(1084229984) Summary: Connection request warnings: 0
(Debug:01)(1084229984) Summary: Connection requests failed: 0 [0/0/0/0/0/0/0/0]
(Debug:01)(1084229984) Summary: Connections valid: 0
(Debug:01)(1084229984) Summary: Connection warnings: 0
(Debug:01)(1084229984) Summary: Connections failed: 0 [0/0/0/0/0/0/0/0]
(Debug:01)(1084229984) Summary: Received 102 frames [1/0/0/100/0/0/1/0/0/0/0], 948136 bytes
(Debug:01)(1084229984) Summary: Input buffer size: 10240 bytes
(Debug:01)(1084229984) Summary: Sent 11 frames [0/1/0/0/0/9/0/1/0/0/0], 1284 bytes
(Debug:01)(1084229984) Summary: Cached Keys: 0
(Debug:01)(1084229984) Summary: ----------------------------
(Debug:01)(1084229984) Summary: Stations with unknown type (yet): 0
(Debug:01)(1084229984) Summary: ----------------------------
(Debug:01)(1084229984) Summary: Thread info: 4/4/0
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38816 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38823 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CRQ_E: HILR_Linux4 172.21.50.58:8000 to 172.21.50.188:38826 CD1.0 0 connection closed unexpected
(Debug:01)(1105209696) CON_I: HILR_Linux4 172.21.50.58:8002 to 172.21.50.188:38832 CD1.0 0 station connected
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1126189408) CON_W: MKAR_Linux4 172.21.50.58:8003 to 172.21.50.188:38576 CD1.1 0 frame not signed
(Debug:01)(1136679264) CON_W: MKAR_Solaris 172.21.60.190:10002 to 172.21.50.188:38578 CD1.1 0 frame not signed
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 4
heartbeat[15600]: 2007/08/08_09:18:39 info: hb_giveup_resources(): current status: active
heartbeat[15600]: 2007/08/08_09:18:39 debug: Sending hold resources msg: none, stable=0 # shutdown
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_rsc_isstable: ResourceMgmt_child_count: 0, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 5
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:39 info: Heartbeat shutdown in progress. (15600)
heartbeat[15600]: 2007/08/08_09:18:39 debug: Process [hb_giveup_resources] started pid 23111
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:39 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:39 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[23111]: 2007/08/08_09:18:39 info: Giving up all HA resources.
ResourceManager[23124]:	2007/08/08_09:18:39 info: Releasing resource group: linux6 172.21.50.188/24/eth0 drbddisk::r0 Filesystem::/dev/drbd0::/data0::ext3 ChangeSrcIP::172.21.50.58::32::eth0::0 ChangeSrcIP::172.21.60.190::32::eth0::0 drbddisk::r1 Filesystem::/dev/drbd1::/data1::ext3 cdrecvd cdsendd
ResourceManager[23124]:	2007/08/08_09:18:39 info: Running /usr/local/etc/ha.d/resource.d/cdsendd  stop
ResourceManager[23124]:	2007/08/08_09:18:39 debug: Starting /usr/local/etc/ha.d/resource.d/cdsendd  stop

ResourceManager[23124]:	2007/08/08_09:18:39 debug: /usr/local/etc/ha.d/resource.d/cdsendd  stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:39 info: Running /usr/local/etc/ha.d/resource.d/cdrecvd  stop
ResourceManager[23124]:	2007/08/08_09:18:39 debug: Starting /usr/local/etc/ha.d/resource.d/cdrecvd  stop

ResourceManager[23124]:	2007/08/08_09:18:39 debug: /usr/local/etc/ha.d/resource.d/cdrecvd  stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:39 info: Running /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 stop
ResourceManager[23124]:	2007/08/08_09:18:39 debug: Starting /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 stop
Filesystem[23240]:	2007/08/08_09:18:39 INFO: Running stop for /dev/drbd1 on /data1
Filesystem[23240]:	2007/08/08_09:18:39 INFO: Trying to unmount /data1
Filesystem[23240]:	2007/08/08_09:18:39 INFO: unmounted /data1 successfully
Filesystem[23229]:	2007/08/08_09:18:39 INFO:  Success
INFO:  Success
ResourceManager[23124]:	2007/08/08_09:18:39 debug: /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd1 /data1 ext3 stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:39 info: Running /usr/local/etc/ha.d/resource.d/drbddisk r1 stop
ResourceManager[23124]:	2007/08/08_09:18:39 debug: Starting /usr/local/etc/ha.d/resource.d/drbddisk r1 stop
ResourceManager[23124]:	2007/08/08_09:18:39 debug: /usr/local/etc/ha.d/resource.d/drbddisk r1 stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:39 info: Running /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 stop
ResourceManager[23124]:	2007/08/08_09:18:39 debug: Starting /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 stop
ChangeSrcIP[23355]:	2007/08/08_09:18:40 INFO: Running stop for managing routes.
ChangeSrcIP[23344]:	2007/08/08_09:18:40 INFO:  Success
INFO:  Success
ResourceManager[23124]:	2007/08/08_09:18:40 debug: /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.60.190 32 eth0 0 stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:40 info: Running /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 stop
ResourceManager[23124]:	2007/08/08_09:18:40 debug: Starting /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 stop
ChangeSrcIP[23417]:	2007/08/08_09:18:40 INFO: Running stop for managing routes.
ChangeSrcIP[23406]:	2007/08/08_09:18:40 INFO:  Success
INFO:  Success
ResourceManager[23124]:	2007/08/08_09:18:40 debug: /usr/local/etc/ha.d/resource.d/ChangeSrcIP 172.21.50.58 32 eth0 0 stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:40 info: Running /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 stop
ResourceManager[23124]:	2007/08/08_09:18:40 debug: Starting /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 stop
Filesystem[23479]:	2007/08/08_09:18:40 INFO: Running stop for /dev/drbd0 on /data0
Filesystem[23479]:	2007/08/08_09:18:40 INFO: Trying to unmount /data0
Filesystem[23479]:	2007/08/08_09:18:40 INFO: unmounted /data0 successfully
Filesystem[23468]:	2007/08/08_09:18:40 INFO:  Success
INFO:  Success
ResourceManager[23124]:	2007/08/08_09:18:40 debug: /usr/local/etc/ha.d/resource.d/Filesystem /dev/drbd0 /data0 ext3 stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:40 info: Running /usr/local/etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[23124]:	2007/08/08_09:18:40 debug: Starting /usr/local/etc/ha.d/resource.d/drbddisk r0 stop
ResourceManager[23124]:	2007/08/08_09:18:40 debug: /usr/local/etc/ha.d/resource.d/drbddisk r0 stop done. RC=0
ResourceManager[23124]:	2007/08/08_09:18:40 info: Running /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 stop
ResourceManager[23124]:	2007/08/08_09:18:40 debug: Starting /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 stop
In IP Stop
SIOCDELRT: No such process
IPaddr[23614]:	2007/08/08_09:18:40 INFO: ifconfig eth0:0 down
IPaddr[23585]:	2007/08/08_09:18:40 INFO:  Success
INFO:  Success
ResourceManager[23124]:	2007/08/08_09:18:40 debug: /usr/local/etc/ha.d/resource.d/IPaddr 172.21.50.188/24/eth0 stop done. RC=0
heartbeat[23111]: 2007/08/08_09:18:40 info: All HA resources relinquished.
heartbeat[23111]: 2007/08/08_09:18:40 debug: Sending T_SHUTDONE.
heartbeat[23111]: 2007/08/08_09:18:40 info: FIFO message [type shutdone] written rc=38
heartbeat[15600]: 2007/08/08_09:18:40 debug: hb_rsc_isstable: ResourceMgmt_child_count: 1, other_is_stable: 1, takeover_in_progress: 0, going_standby: 0, standby running(ms): 0, resourcestate: 5
heartbeat[15600]: 2007/08/08_09:18:40 debug: Received T_SHUTDONE from us.
heartbeat[15600]: 2007/08/08_09:18:40 debug: Calling hb_mcp_final_shutdown in a second.
heartbeat[15600]: 2007/08/08_09:18:40 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:40 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:40 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:40 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:40 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:40 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:40 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:40 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:40 debug: RscMgmtProc 'hb_giveup_resources' exited code 0
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_mcp_final_shutdown() phase 0
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_mcp_final_shutdown()- immediate completion.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_mcp_final_shutdown() phase 1
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Cannot write to media pipe 1: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Cannot write to media pipe 2: Resource temporarily unavailable
heartbeat[15600]: 2007/08/08_09:18:41 ERROR: Shutting down.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown() called.
heartbeat[15600]: 2007/08/08_09:18:41 debug: hb_initiate_shutdown(): shutdown already in progress
heartbeat[15600]: 2007/08/08_09:18:42 debug: hb_mcp_final_shutdown() phase 2
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBREAD process 15609 with signal 15
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: glib: Unable to send bcast [-1] packet(len=203): Interrupted system call
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBWRITE process 15610 with signal 15
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG: Dumping message with 9 fields
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: glib: Unable to send bcast [-1] packet(len=203): Interrupted system call
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[0] : [t=NS_ackmsg]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[1] : [dest=linux7]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG: Dumping message with 9 fields
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[2] : [ackseq=5706]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[0] : [t=NS_ackmsg]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[3] : [(1)destuuid=0x5ef8b8(37 28)]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[1] : [dest=linux7]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[4] : [src=linux7]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[2] : [ackseq=5706]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[5] : [(1)srcuuid=0x5ef9f8(36 27)]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[3] : [(1)destuuid=0x5ef7d8(37 28)]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[6] : [hg=46b83d21]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[4] : [src=linux7]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[7] : [ts=46b917ab]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[5] : [(1)srcuuid=0x5ef918(36 27)]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: MSG[8] : [ttl=4]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[6] : [hg=46b83d21]
heartbeat[15608]: 2007/08/08_09:18:42 ERROR: write failure on bcast eth2.: Interrupted system call
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[7] : [ts=46b917ab]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: MSG[8] : [ttl=4]
heartbeat[15608]: 2007/08/08_09:18:42 debug: Process 15608 processing SIGTERM
heartbeat[15608]: 2007/08/08_09:18:42 debug: Exiting from pid 15608 [rc=15]
heartbeat[15606]: 2007/08/08_09:18:42 ERROR: write failure on bcast eth1.: Interrupted system call
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBREAD process 15611 with signal 15
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBFIFO process 15603 with signal 15
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBWRITE process 15604 with signal 15
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBREAD process 15605 with signal 15
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBWRITE process 15606 with signal 15
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBREAD process 15607 with signal 15
heartbeat[15600]: 2007/08/08_09:18:42 info: killing HBWRITE process 15608 with signal 15
heartbeat[15603]: 2007/08/08_09:18:42 debug: Process 15603 processing SIGTERM
heartbeat[15603]: 2007/08/08_09:18:42 debug: Exiting from pid 15603 [rc=15]
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15608 exited. 9 remaining
heartbeat[15611]: 2007/08/08_09:18:42 debug: Process 15611 processing SIGTERM
heartbeat[15611]: 2007/08/08_09:18:42 debug: Exiting from pid 15611 [rc=15]
heartbeat[15604]: 2007/08/08_09:18:42 debug: Process 15604 processing SIGTERM
heartbeat[15604]: 2007/08/08_09:18:42 debug: Calling localdie() function
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15603 exited. 8 remaining
heartbeat[15604]: 2007/08/08_09:18:42 debug: glib: serial_localdie: Flushing tty
heartbeat[15604]: 2007/08/08_09:18:42 debug: Exiting from pid 15604 [rc=15]
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15611 exited. 7 remaining
heartbeat[15605]: 2007/08/08_09:18:42 debug: Process 15605 processing SIGTERM
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15604 exited. 6 remaining
heartbeat[15605]: 2007/08/08_09:18:42 debug: Exiting from pid 15605 [rc=15]
heartbeat[15606]: 2007/08/08_09:18:42 debug: Process 15606 processing SIGTERM
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15605 exited. 5 remaining
heartbeat[15607]: 2007/08/08_09:18:42 debug: Process 15607 processing SIGTERM
heartbeat[15607]: 2007/08/08_09:18:42 debug: Exiting from pid 15607 [rc=15]
heartbeat[15606]: 2007/08/08_09:18:42 debug: Exiting from pid 15606 [rc=15]
heartbeat[15609]: 2007/08/08_09:18:42 debug: Process 15609 processing SIGTERM
heartbeat[15609]: 2007/08/08_09:18:42 debug: Exiting from pid 15609 [rc=15]
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15607 exited. 4 remaining
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15606 exited. 3 remaining
heartbeat[15610]: 2007/08/08_09:18:42 debug: Process 15610 processing SIGTERM
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15609 exited. 2 remaining
heartbeat[15610]: 2007/08/08_09:18:42 debug: Exiting from pid 15610 [rc=15]
heartbeat[15600]: 2007/08/08_09:18:42 info: Core process 15610 exited. 1 remaining
heartbeat[15600]: 2007/08/08_09:18:42 info: linux7 Heartbeat shutdown complete.
heartbeat[15600]: 2007/08/08_09:18:42 debug: Exiting from pid 15600 [rc=0]
