The logs in "messages" indicates that the VCS did not receive the GAB heartbeat
for 15 seconds due to which the system was forcefully crashed (and noe
rebooted) by VCS.
You would have noticed that after the system came back online, it would still
be out of cluster (i.e., in "not seeded" state).
- Suraj
________________________________
From: Aleksandr Nepomnyashchiy <anep...@gmail.com>
To: veritas-ha@mailman.eng.auburn.edu
Sent: Thursday, 2 April, 2009 2:19:50 AM
Subject: [Veritas-ha] Did VCS rebooted the cluster node? Please help
interpreting the log
Dear VCS gurus,
Please help interpreting the log - did system rebooted due to the VCS
error ? Or VCS rebooted the system?
I am trying to make sense of the logs... "libthread panic: cannot
create new lwp : dumping core" from the VCS engine_A.log seems
suspicious.
r...@node_p /var/adm #
r...@node_p /var/adm # more messages
Apr 1 14:32:10 node_p unix: GAB:20057: Port h process 543 inactive 13 sec
Apr 1 14:32:11 node_p unix: GAB:20057: Port h process 543 inactive 14 sec
Apr 1 14:32:12 node_p unix: GAB:20058: Port h process 543: heartbeat
failed, killing process
Apr 1 14:32:12 node_p unix: GAB:20059: Port h heartbeat interval
15000 msec. Statistics:
Apr 1 14:32:12 node_p unix: GAB: Port h: heartbeats in 0 ~ 3000 msec: 56500211
Apr 1 14:32:12 node_p unix: GAB: Port h: heartbeats in 3000 ~ 6000 msec: 0
Apr 1 14:32:12 node_p unix: GAB: Port h: heartbeats in 6000 ~ 9000 msec: 0
Apr 1 14:32:12 node_p unix: GAB: Port h: heartbeats in 9000 ~ 12000 msec: 0
Apr 1 14:32:12 node_p unix: GAB: Port h: heartbeats in 12000 ~ 15000 msec: 0
Apr 1 14:32:12 node_p unix: GAB:20062: process 543, command line: had
Apr 1 14:32:12 node_p unix: GAB:20064: state: SRUN (running)
Apr 1 14:32:12 node_p unix: GAB:20070: total lwp: 1
Apr 1 14:32:12 node_p unix: GAB:20071: total lwp swapped out: 0
Apr 1 14:32:12 node_p unix: GAB:20072: lwp information:
Apr 1 14:32:12 node_p unix: GAB:20073: 1: thread id 1,
kthread at 300030cb4e0, in system call 5
Apr 1 14:32:12 node_p unix: GAB:20075: TS_SLEEP (waiting an event)
Apr 1 14:32:12 node_p unix: GAB:20088: System information:
Apr 1 14:32:12 node_p unix: GAB:20089: number of cpu: 2
Apr 1 14:32:12 node_p unix: GAB:20090: physical memory:
2043816 K
Apr 1 14:32:12 node_p unix: GAB:20091: free memory:
54608 K
Apr 1 14:32:12 node_p unix: GAB:20092: average free memory in
5 sec: 54232 K
Apr 1 14:32:12 node_p unix: GAB:20093: average free memory in
30 sec: 54184 K
Apr 1 14:32:12 node_p unix: GAB:20094: number of processes: 58
Apr 1 14:32:12 node_p unix: GAB:20095: load average in 1 min:
420
Apr 1 14:32:12 node_p unix: GAB:20096: load average in 5 min:
449
Apr 1 14:32:12 node_p unix: GAB:20097: load average in 15 min:
465
Apr 1 14:32:12 node_p unix: GAB:20098: pagein rate: 0
Apr 1 14:32:12 node_p unix: GAB:20099: pageout rate: 0
Apr 1 14:32:12 node_p unix: GAB:20041: Port h: client process
failure: killing process
Apr 1 14:32:27 node_p unix: GAB:20035: Port h attempting to kill
process due to client process failure
Apr 1 14:33:12 node_p last message repeated 3 times
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ee790) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0xb 0x9c 0xea 0x4 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x1861
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x436608) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0x3d 0xf3
0x1e 0x0 0x0 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x437208) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x1b 0x78 0x6c 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x436788) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0xb 0x9c 0x5d 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x436008) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x8 0xb9 0x6c 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ef210) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0xb 0x9c 0x8d 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x4aa0040) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x1f 0xcc 0x77 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x4aa1b40) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x1f 0xcc 0x78 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x436488) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x5 0xd7 0x3c 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x4aa01c0) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0x24 0x1f
0x4b 0x0 0x0 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24efb10) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x1f 0xcb 0x8a 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24eec10) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0xb 0x9d 0x1e 0x2 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x437088) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0x32 0x6a
0x74 0x0 0x0 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x437388) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0xb 0x9b 0xe9 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ef090) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0xd 0xd 0x1b 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x4aa0940) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0x5e 0x1b
0x33 0x0 0x0 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ef390) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0xdb 0x28
0x30 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24efe10) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0x60 0xd5
0x70 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ef810) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0xa 0x0 0xc2 0xc7 0x1 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ef690) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x28 0x0 0x0 0xd3 0x7f
0x30 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x8e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x437508) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x28 0x0 0x0 0x41 0x69
0xa8 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x8e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x436308) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x28 0x0 0x1 0xc 0x54
0x30 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x8e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ef990) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x28 0x0 0x0 0x90 0x3e
0x40 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x8e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x24ee190) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0xd3 0x93
0x70 0x0 0x0 0x4 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Cmd (0x436f08) dump for Target 1 Lun 0:
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p cdb=[ 0x2a 0x0 0x0 0xd3 0xcf
0x50 0x0 0x0 0x10 0x0 ]
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_flags=0x4000 pkt_statistics=0x61
pkt_state=0x7
Apr 1 14:33:42 node_p unix: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p pkt_scbp=0x0 cmd_flags=0x18e1
Apr 1 14:33:42 node_p unix: WARNING: /p...@1f,4000/s...@3 (glm0):
Apr 1 14:33:42 node_p Disconnected tagged cmd(s) (25) timeout
for Target 1.0
Apr 1 14:33:42 node_p unix: WARNING: ID[SUNWpd.glm.cmd_timeout.6018]
Apr 1 14:33:42 node_p unix: WARNING: /p...@1f,4000/s...@3/s...@1,0 (sd1):
Apr 1 14:33:42 node_p SCSI transport failed: reason
'timeout': retrying command
Apr 1 14:33:42 node_p unix: panic[cpu0]/thread=2a10004dd60:
Apr 1 14:33:42 node_p unix: GAB: Port h halting system due to client
process failure
Apr 1 14:33:42 node_p unix:
Apr 1 14:33:42 node_p unix: syncing file systems...
Apr 1 14:33:46 node_p unix: WARNING: /p...@1f,4000/s...@3/s...@1,0 (sd1):
Apr 1 14:33:46 node_p Error for Command: write
Error Level: Retryable
Apr 1 14:33:46 node_p unix: Requested Block: 761066
Error Block: 761066
Apr 1 14:33:46 node_p unix: Vendor: IBM
Serial Number: 00331CQ163
Apr 1 14:33:46 node_p unix: Sense Key: Unit Attention
Apr 1 14:33:46 node_p unix: ASC: 0x29 (<vendor unique code 0x29>),
ASCQ: 0x3, FRU: 0x0
Apr 1 14:33:46 node_p unix: 3
Apr 1 14:33:47 node_p unix: 2
Apr 1 14:33:48 node_p unix: done
Apr 1 14:33:49 node_p unix: dumping to /dev/dsk/c0t0d0s1, offset 65536
Apr 1 14:34:14 node_p unix: ^M100% done: 32710 pages dumped,
compression ratio 3.15,
Apr 1 14:34:14 node_p unix: dump succeeded
Apr 1 14:35:41 node_p unix: ^MSunOS Release 5.7 Version
Generic_106541-46 64-bit [UNIX(R) System V Release 4.0]
Apr 1 14:35:41 node_p unix: Copyright (c) 1983-2002, Sun Microsystems, Inc.
Apr 1 14:35:41 node_p unix: Ethernet address = 8:0:20:f0:18:f4
Apr 1 14:35:41 node_p unix: WARNING: forceload of drv/scsi failed
Apr 1 14:35:41 node_p unix: mem = 2097152K (0x80000000)
Apr 1 14:35:41 node_p unix: avail mem = 2061238272
Apr 1 14:35:41 node_p unix: root nexus = Sun Enterprise 420R (2 X
UltraSPARC-II 450MHz)
Apr 1 14:35:41 node_p unix: pci0 at root: UPA 0x1f 0x4000
Apr 1 14:35:41 node_p unix: pci0 is /p...@1f,4000
Apr 1 14:35:41 node_p unix: pci1 at root: UPA 0x1f 0x2000
Apr 1 14:35:41 node_p unix: pci1 is /p...@1f,2000
r...@node_p /var/adm #
============ This is a fragment of engine_A.log from the ============
TAG_B 2009/03/06 05:08:20 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/MultiNICA/monitor) for resource
(my_nic) was abnormally terminated with the exit code (86).
TAG_B 2009/03/10 15:01:58 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/DiskGroup/monitor) for resource
(my_dg) was abnormally terminated with the exit code (86).
TAG_B 2009/03/21 19:19:54 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/DiskGroup/monitor) for resource
(my_dg) was abnormally terminated with the exit code (86).
TAG_B 2009/03/21 22:02:48 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/IPMultiNIC/monitor) for resource
(my_ip) was abnormally terminated with the exit code (86).
TAG_B 2009/03/24 18:07:51 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/DiskGroup/monitor) for resource
(my_dg) was abnormally terminated with the exit code (86).
TAG_B 2009/03/24 21:15:51 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/DiskGroup/monitor) for resource
(my_dg) was abnormally terminated with the exit code (86).
TAG_B 2009/03/26 02:57:12 (node_p)
VCS:13039:Program(/opt/VRTSvcs/bin/MultiNICA/monitor) for resource
(my_nic) was abnormally terminated with the exit code (86).
TAG_E 2009/03/31 20:03:43 (node_p) VCS:13001:Output of the completed
operation (monitor) on resource (my_ip)
libthread panic: cannot create new lwp : dumping core (PID: 24953 LWP 2)
TAG_E 2009/04/01 14:33:43 VCS:10077:received new cluster membership
Thank you,
Aleksandr
_______________________________________________
Veritas-ha maillist - Veritas-ha@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-ha
Get perfect Email ID for your Resume. Grab now
http://in.promos.yahoo.com/address_______________________________________________
Veritas-ha maillist - Veritas-ha@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-ha
