First of all, thanks for reading my post.
We are facing a problem in a Solaris10-sparc box. Our program freezes (stop
sending sctp messages). We think there may be a problem with the driver because
it doesn't die even after a kill -9 and the call stack shows two thread waiting
inside sctp routines.
Any suggestion will be really appreciated. Thanks in advance.
Details
-------
1) The program is running and exchanging messages. It uses a multihomed
association with two network cards.
2) The wires are unplugged from both cards.
3) After 30 minutes, the wires are plugged again.
4) The program starts exchanging messages again.
5) It freezes.
6) kill -9
7) The program doesn't die completely. ps -ef shows it, but pstack, pmap and
gcore reports 'no such process'.
8) echo '0t2777 ::pid2proc | ::walk thread | ::findstack ' | mdb -k
stack pointer for thread 30005918500: 2a100dbaf11
[ 000002a100dbaf11 cv_wait+0x38() ]
000002a100dbafc1 exitlwps+0x11c()
000002a100dbb071 proc_exit+0x20()
000002a100dbb121 exit+8()
000002a100dbb1d1 post_syscall+0x40c()
000002a100dbb2e1 syscall_trap+0x16c()
stack pointer for thread 3000ce53b00: 2a100496d01
[ 000002a100496d01 cv_wait+0x38() ]
000002a100496db1 sctp_sendmsg+0x2ac()
000002a100496e81 sosctp_sendmsg+0x320()
000002a100496f41 sendit+0x134()
000002a100497011 sendmsg+0x294()
000002a1004972e1 syscall_trap+0xac()
stack pointer for thread 3000cdd67a0: 2a1004bed01
[ 000002a1004bed01 cv_wait+0x38() ]
000002a1004bedb1 sctp_sendmsg+0x2ac()
000002a1004bee81 sosctp_sendmsg+0x320()
000002a1004bef41 sendit+0x134()
000002a1004bf011 sendmsg+0x294()
000002a1004bf2e1 syscall_trap+0xac()
>
9) Our program establishes 2 associations
9.1) First association
(local) (201.70.251.131:2972, 201.70.251.163:2972) ==> (remote)
(201.12.231.39:2972, 201.12.231.38:2972)
9.2) Second association
(local) (201.70.251.131:2932,201.70.251.163:2932) ==> (remote)
(201.12.231.37:2932, 201.12.231.38:2932 )
$ snoop -t a -r -i sniff.pcap
#
# at this time both association are established
#
1 15:18:23.78305 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
2 15:18:23.80236 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
3 15:18:26.72304 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
4 15:18:26.75429 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
5 15:18:30.11306 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
6 15:18:30.13243 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
7 15:18:30.61338 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
8 15:18:32.99304 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
9 15:18:33.02442 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
10 15:18:35.82305 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
11 15:18:35.84257 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
12 15:18:38.55306 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
13 15:18:38.58430 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
14 15:18:42.16305 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
15 15:18:42.18250 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
16 15:18:44.85303 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
17 15:18:44.88452 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
18 15:18:48.26306 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
19 15:18:48.28250 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
20 15:18:48.41544 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
3fe1 str 0/1498 ppid 3
21 15:18:50.57304 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
22 15:18:50.60437 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
23 15:18:54.30303 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
24 15:18:54.32253 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
25 15:18:56.32303 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
26 15:18:56.45447 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
27 15:18:58.77303 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
28 15:18:58.78355 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
29 15:19:0.37305 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
30 15:19:0.39255 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
31 15:19:1.05304 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
32 15:19:1.06559 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
33 15:19:2.63304 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
34 15:19:2.66459 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
35 15:19:2.71358 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
36 15:19:2.71368 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
ACK
37 15:19:4.11368 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
7346 str 0/1509 ppid 3
38 15:19:4.11426 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 SACK tsn
7346 win 102400 gaps/dups 0/0 Data tsn dfae7
f41 str 0/1240 ppid 3
39 15:19:4.21365 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 SACK tsn
dfae7f41 win 1240320 gaps/dups 0/0
40 15:19:6.77302 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
41 15:19:6.80259 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
42 15:19:6.86552 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
43 15:19:6.86559 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
ACK
44 15:19:7.31303 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
45 15:19:7.32568 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
46 15:19:8.47302 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
47 15:19:8.50454 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
48 15:19:10.03307 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
49 15:19:10.04361 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
50 15:19:12.70304 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
51 15:19:12.72266 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
52 15:19:12.95303 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
53 15:19:12.96585 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
54 15:19:14.76302 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
55 15:19:14.81476 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
56 15:19:15.74302 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
57 15:19:15.75376 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
58 15:19:18.66303 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
59 15:19:18.67585 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
60 15:19:18.76370 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
61 15:19:18.76379 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
ACK
62 15:19:19.01305 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
63 15:19:19.03274 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
64 15:19:20.36303 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
65 15:19:20.39466 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
66 15:19:21.54305 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
67 15:19:21.55387 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
68 15:19:22.91582 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
69 15:19:22.91595 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
ACK
70 15:19:25.08302 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
71 15:19:25.09574 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
72 15:19:25.42302 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
73 15:19:25.50262 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
74 15:19:26.01303 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
75 15:19:26.04471 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
76 15:19:27.68303 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
77 15:19:27.69386 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
78 15:19:31.36301 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
79 15:19:31.37583 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
80 15:19:31.91304 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
81 15:19:31.98278 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
82 15:19:32.51302 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
83 15:19:32.54484 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
84 15:19:33.57303 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
85 15:19:33.58395 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
86 15:19:34.81392 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
87 15:19:34.81404 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
ACK
88 15:19:37.07301 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
89 15:19:37.08583 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
90 15:19:38.47301 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
91 15:19:38.49292 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
92 15:19:38.59298 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
93 15:19:38.62485 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
94 15:19:38.96603 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
95 15:19:38.96615 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
ACK
96 15:19:39.82301 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
97 15:19:39.83389 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
98 15:19:43.45302 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
99 15:19:43.46589 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
100 15:19:44.43302 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
101 15:19:44.45270 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
102 15:19:44.72298 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
103 15:19:44.78498 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
104 15:19:46.09302 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
105 15:19:46.10399 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
106 15:19:48.41600 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
3fe2 str 0/1499 ppid 3
107 15:19:48.41615 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 SACK tsn
3fe2 win 102392 gaps/dups 0/0
108 15:19:48.41649 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Data tsn
756cbe17 str 0/1241 ppid 3
109 15:19:48.51597 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 SACK tsn
756cbe17 win 1240320 gaps/dups 0/0
110 15:19:50.31302 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
111 15:19:50.45514 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
112 15:19:50.86404 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
113 15:19:50.86414 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
ACK
114 15:19:50.92298 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
115 15:19:50.94300 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
116 15:19:52.55300 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
117 15:19:52.56401 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
118 15:19:54.09303 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
119 15:19:54.10602 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
120 15:19:55.01602 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
121 15:19:55.01615 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
ACK
122 15:19:56.80301 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
123 15:19:56.91296 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
124 15:19:56.93290 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
125 15:19:56.93529 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
126 15:19:58.26301 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
127 15:19:58.27415 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
128 15:20:0.36301 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
129 15:20:0.37611 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
130 15:20:2.63301 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
131 15:20:2.66515 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
132 15:20:3.26301 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
133 15:20:3.30296 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
134 15:20:4.11411 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
7347 str 0/1510 ppid 3
135 15:20:4.11427 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 SACK tsn
7347 win 102392 gaps/dups 0/0
136 15:20:4.11466 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Data tsn
dfae7f42 str 0/1241 ppid 3
137 15:20:4.21422 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 SACK tsn
dfae7f42 win 1240320 gaps/dups 0/0
138 15:20:6.46300 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
139 15:20:6.47626 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
140 15:20:6.91414 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
141 15:20:6.91427 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
ACK
142 15:20:8.33302 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
143 15:20:8.36516 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
144 15:20:9.70302 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
145 15:20:9.78301 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
146 15:20:10.47302 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
147 15:20:10.48417 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
148 15:20:11.02610 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
149 15:20:11.02625 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
ACK
150 15:20:12.60302 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
151 15:20:12.61611 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
152 15:20:14.26302 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
153 15:20:14.29536 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
154 15:20:16.26300 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
155 15:20:16.28313 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
156 15:20:16.87298 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
157 15:20:16.88421 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
158 15:20:18.28300 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
159 15:20:18.29627 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
160 15:20:20.16301 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
161 15:20:20.19530 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
162 15:20:21.27635 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
3fe3 str 0/1500 ppid 3
163 15:20:21.37298 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 SACK tsn
3fe3 win 102400 gaps/dups 0/0
164 15:20:21.37432 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
7348 str 0/1511 ppid 3
165 15:20:21.47298 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 SACK tsn
7348 win 102400 gaps/dups 0/0
166 15:20:21.95300 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
167 15:20:21.97299 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
168 15:20:22.92425 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
169 15:20:22.92437 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
ACK
170 15:20:23.38301 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
171 15:20:23.39423 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
172 15:20:24.79302 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
173 15:20:24.80627 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
174 15:20:25.97297 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
175 15:20:26.09539 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
176 15:20:27.07621 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
177 15:20:27.07635 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
ACK
178 15:20:27.90299 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
179 15:20:27.92305 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
180 15:20:28.96297 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
181 15:20:28.97441 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
182 15:20:30.65298 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
183 15:20:30.66636 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
184 15:20:32.45300 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
185 15:20:32.57541 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
186 15:20:34.36300 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
187 15:20:34.38323 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
ACK
188 15:20:35.04301 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
189 15:20:35.05443 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
190 15:20:36.29298 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
191 15:20:36.30639 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
192 15:20:38.74299 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
193 15:20:38.78536 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
ACK
194 15:20:40.38297 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
195 15:20:41.10299 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
196 15:20:41.11445 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
197 15:20:41.38297 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
198 15:20:41.99297 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
199 15:20:42.00639 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
#
# We think this is the moment when the wires were unplugged
#
200 15:20:43.39298 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
201 15:20:45.06298 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
202 15:20:46.06301 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
203 15:20:46.40297 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
204 15:20:46.77295 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
205 15:20:47.69299 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
206 15:20:47.77294 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
207 15:20:48.07298 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
208 15:20:48.70296 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
209 15:20:49.41297 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Heartbeat
210 15:20:49.77604 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
8c39430e str 2/2 win 102400
211 15:20:50.70297 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
212 15:20:51.07298 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Heartbeat
213 15:20:52.77298 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
8c39430e str 2/2 win 102400
214 15:20:53.72667 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
60c234d8 str 2/2 win 102400
215 15:20:56.72298 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
60c234d8 str 2/2 win 102400
216 15:20:58.77297 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
8c39430e str 2/2 win 102400
217 15:21:2.72299 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
60c234d8 str 2/2 win 102400
218 15:21:10.77297 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
8c39430e str 2/2 win 102400
219 15:21:14.72297 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
60c234d8 str 2/2 win 102400
220 15:21:34.77296 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
8c39430e str 2/2 win 102400
221 15:21:38.72296 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
60c234d8 str 2/2 win 102400
222 15:22:22.77293 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
8c39430e str 2/2 win 102400
223 15:22:26.72295 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
60c234d8 str 2/2 win 102400
224 15:23:22.77763 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
e88d7f5 str 2/2 win 102400
225 15:23:25.77290 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
e88d7f5 str 2/2 win 102400
226 15:23:26.72351 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
d46ecd62 str 2/2 win 102400
227 15:23:29.72290 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
d46ecd62 str 2/2 win 102400
228 15:23:31.77292 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
e88d7f5 str 2/2 win 102400
229 15:23:35.72290 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
d46ecd62 str 2/2 win 102400
230 15:23:43.77290 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
e88d7f5 str 2/2 win 102400
231 15:23:47.72289 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
d46ecd62 str 2/2 win 102400
232 15:24:7.77289 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
e88d7f5 str 2/2 win 102400
233 15:24:11.72288 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
d46ecd62 str 2/2 win 102400
234 15:24:55.77288 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
e88d7f5 str 2/2 win 102400
235 15:24:59.72286 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
d46ecd62 str 2/2 win 102400
236 15:25:55.77509 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
64c17f0d str 2/2 win 102400
237 15:25:58.77283 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
64c17f0d str 2/2 win 102400
238 15:25:59.72503 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
3f7a9636 str 2/2 win 102400
239 15:26:2.72283 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
3f7a9636 str 2/2 win 102400
240 15:26:4.77283 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
64c17f0d str 2/2 win 102400
241 15:26:8.72283 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
3f7a9636 str 2/2 win 102400
242 15:26:16.77281 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
64c17f0d str 2/2 win 102400
243 15:26:20.72282 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
3f7a9636 str 2/2 win 102400
244 15:26:40.77282 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
64c17f0d str 2/2 win 102400
245 15:26:44.72281 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
3f7a9636 str 2/2 win 102400
246 15:27:28.77278 201.70.251.131 -> 201.36.200.37 SCTP D=2932 S=2932 Init tsn
64c17f0d str 2/2 win 102400
247 15:27:32.72278 201.70.251.131 -> 201.12.231.38 SCTP D=2972 S=2972 Init tsn
3f7a9636 str 2/2 win 102400
248 15:28:28.77332 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
a9a1330a str 2/2 win 102400
249 15:28:31.77275 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
a9a1330a str 2/2 win 102400
250 15:28:32.72333 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
882c1ce str 2/2 win 102400
251 15:28:35.72275 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
882c1ce str 2/2 win 102400
252 15:28:37.77274 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
a9a1330a str 2/2 win 102400
253 15:28:41.72275 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
882c1ce str 2/2 win 102400
254 15:28:49.77274 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
a9a1330a str 2/2 win 102400
255 15:28:53.72276 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
882c1ce str 2/2 win 102400
#
# 26 minutes of silence ...
# first incoming packet ==> wires plugged again?
#
256 15:54:7.73514 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Init tsn
601acb2a str 2/2 win 102400
257 15:54:7.75270 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Init ACK
tsn 67ba str 2/17 win 1240320
258 15:54:7.75285 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Cookie
259 15:54:7.77229 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Init tsn
6a776d31 str 2/2 win 102400
260 15:54:7.77240 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Cookie ACK
261 15:54:7.77491 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Data tsn
601acb2a str 0/0 ppid 3
262 15:54:7.78973 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Init ACK
tsn 173c str 2/17 win 1240320
263 15:54:7.78982 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Cookie
264 15:54:7.79253 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 SACK tsn
601acb2a win 1240256 gaps/dups 0/0
265 15:54:7.79259 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Heartbeat
266 15:54:7.80253 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
ACK
267 15:54:7.80943 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Cookie ACK
268 15:54:7.81174 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Data tsn
6a776d31 str 0/0 ppid 3
269 15:54:7.82943 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 SACK tsn
6a776d31 win 1240256 gaps/dups 0/0
270 15:54:7.82949 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Heartbeat
271 15:54:7.83942 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
ACK
272 15:54:8.78491 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 Data tsn
601acb2b str 0/1 ppid 3
273 15:54:8.82098 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 Data tsn
6a776d32 str 0/1 ppid 3
274 15:54:8.89245 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Data tsn
67ba str 0/0 ppid 3 Data tsn 67bb str 0/1 pp
id 3 Data tsn 67bc str 0/2 ppid 3
275 15:54:8.89255 201.70.251.163 -> 201.12.231.39 SCTP D=2972 S=2972 SACK tsn
67bc win 102360 gaps/dups 0/0
276 15:54:8.89962 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Data tsn
173c str 0/0 ppid 3 Data tsn 173d str 0/1 pp
id 3 Data tsn 173e str 0/2 ppid 3
277 15:54:8.89971 201.70.251.163 -> 201.36.200.38 SCTP D=2932 S=2932 SACK tsn
173e win 102360 gaps/dups 0/0
#
# 18 seconds of silence ...
# what may be happening here? heartbeats are not acknowlegded anymore
#
278 15:54:24.94268 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
279 15:54:24.94972 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
280 15:54:25.99640 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
281 15:54:26.00414 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
282 15:54:28.04260 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
283 15:54:28.04963 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
284 15:54:31.10422 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
285 15:54:31.18630 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
286 15:54:34.14262 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
287 15:54:34.14964 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
288 15:54:37.20423 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
289 15:54:37.20633 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
290 15:54:40.24278 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
291 15:54:40.24979 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
292 15:54:41.01430 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
293 15:54:43.04277 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
294 15:54:43.04964 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
295 15:54:46.10439 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
296 15:54:46.10633 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
297 15:54:49.14290 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
298 15:54:49.14979 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
299 15:54:52.20438 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
300 15:54:52.20634 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
301 15:54:55.24278 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Heartbeat
302 15:54:55.24980 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Heartbeat
303 15:54:58.04299 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
304 15:54:58.04998 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
305 15:55:1.10458 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
306 15:55:1.10655 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
307 15:55:4.14298 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
308 15:55:4.14985 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
309 15:55:7.20459 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
310 15:55:7.20654 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
311 15:55:7.79295 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
67bd str 0/3 ppid 3
312 15:55:7.84996 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
173f str 0/3 ppid 3
313 15:55:10.20667 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
314 15:55:10.25465 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
315 15:55:10.80304 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
67bd str 0/3 ppid 3
316 15:55:10.85991 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
173f str 0/3 ppid 3
317 15:55:13.26463 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
318 15:55:13.30669 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
319 15:55:13.86466 201.12.231.38 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
67bd str 0/3 ppid 3
320 15:55:13.90660 201.36.200.37 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
173f str 0/3 ppid 3
321 15:55:16.05310 201.12.231.39 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
322 15:55:16.05996 201.36.200.38 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
323 15:55:16.90304 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
67bd str 0/3 ppid 3
324 15:55:16.95991 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
173f str 0/3 ppid 3
325 15:55:19.10667 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
326 15:55:19.11469 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
327 15:55:19.95312 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
67bd str 0/3 ppid 3
328 15:55:20.01000 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
173f str 0/3 ppid 3
329 15:55:22.16478 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
330 15:55:22.16673 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
331 15:55:23.00305 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Data tsn
67bd str 0/3 ppid 3
332 15:55:23.05993 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Data tsn
173f str 0/3 ppid 3
333 15:55:25.21487 201.12.231.38 -> 201.70.251.163 SCTP D=2972 S=2972 Heartbeat
334 15:55:25.21653 201.36.200.37 -> 201.70.251.163 SCTP D=2932 S=2932 Heartbeat
335 15:55:26.05315 201.12.231.39 -> 201.70.251.131 SCTP D=2972 S=2972 Abort
336 15:55:26.11000 201.36.200.38 -> 201.70.251.131 SCTP D=2932 S=2932 Abort
--
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list
networking-discuss@opensolaris.org
