On Fri, Oct 20, 2000 at 04:26:49PM -0500, Dave Dykstra wrote:
> Andrew Tridgell wrote:
> > Dave Dykstra wrote:
> > > The linux server says
> > >
> > > Proto Recv-Q Send-Q Local Address Foreign Address State
> > > tcp 35230 40686 expbuild.research.:8664 dynamic.ih.lucent:36352
>ESTABLISHED
> > >
> > > and the solaris client says
> > >
> > > Local Address Remote Address Swind Send-Q
>Rwind Recv-Q State
> > > dynamic.ih.lucent.com.36352 expbuild.research.bell-labs.com.8664 0 1459
> 8760 0 ESTABLISHED
> > The next thing you have to do is run a sniffer to determine whether it
> > is a Solaris or Linux bug. My bet is this will be the same Linux bug
> > we have observed here. You'll see the Linux box sending data outside
> > the window that the Solaris box is offering, the Solaris box will
> > reject that data by sending a ack with the current window and the
> > Linux box will ignore the hint.
>
> ...
>
> On Fri, Oct 20, 2000 at 10:44:04AM +1000, Andrew Tridgell wrote:
> ...
> > Stephen tells me that a patch went into the 2.2.17 Linux kernel that
> > was supposed to fix this particular problem. If you get a chance it
> > would be worth trying that kernel (or a 2.2.18preX) to see if it
> > solves your problem.
>
> We upgraded to 2.2.17 but it didn't help, unfortunately. An SGI machine in
> the same machine room doesn't exhibit the problem with --timeout 0, though,
> so you're probably right. I didn't get a chance to do any sniffing but
> I'll try doing that later I guess.
I and a colleague have now done some sniffing and it does indeed look like
a Linux bug, but not the one you describe. In fact, we once saw just the
opposite where the Solaris box sent data outside the window the Linux box
was offering, and the Linux box ignored it. We don't think that's necessarily
an error; we figured the machine that sends the extra data may be in the
process of sending an ack anyway and figure since it's got data to send it
may as well send it in the hopes that the window on the receiver will have
opened up in the meantime.
The problem we see is that when the Linux machine's *receive* queue gets
quite full and rsync isn't ready to receive it, the Linux machine seems
unable to send data out of its *send* queue; the tcpdump (attached) shows
that Solaris has told Linux that it's ready to receive more but Linux
doesn't send it. I figure that since the Solaris machine is the receiver
and it has the two rsync processes, the rsync process that's generating the
checksums for the receive side is blasting a lot of data at the single
sending rsync process on the Linux machine which isn't ready to accept it
yet because it is sending a lot of data in the other direction. The rsync
sender would eventually get to read the receive data if Linux would just
transmit its send data.
Do you know who worked on the patch for 2.2.17? I'd like to get this
information to them.
We have reproduced this problem between Linux 2.2.17 & Solaris 7 on a LAN,
in fact two machines on the same etherswitch. We have also tried it between
Linux 2.2.18preX and the same Solaris 7 machine, over a WAN. We didn't see
a problem between the two Linux machines. The tcpdump shows the 2.2.17
Linux machine called "static" which is analagous to "expbuild" in the
above netstat.
For somebody else to reproduce this they should be able to first populate
a directory on a Solaris machine with the data from
rsync://www.bell-labs.com/wwexptools/rsynctst/m17/
and then offer the data that is now at
rsync://www.bell-labs.com/wwexptools/rsynctst/m18/
from an rsync 2.4.6 daemon on a Linux machine and then initiate an rsync
2.4.6 client on the Solaris machine to pull it down on top of the m17
data. We're going to try to work on a smaller test program to reproduce
this.
> ...
> > > This morning I observed that while one client process was working hard for
> > > a long time, the other one was indeed idle a lot of the time so I am again
> > > leaning toward the necessity of Neil Schellenberger's timeout fix. The
> > > above test was run with --timeout 0.
> >
> > Neil's analysis is quite plausible and worth looking into but that is
> > most definately not what is causing the hang you see here.
>
> I'm thinking that this is the likely cause of most of my timeouts with
> large updates, other than the Linux freeze-up. I will try to do more
> testing, and I think it would be very helpful if you could scrutinize his
> patch to see if you think it is a good solution.
I have done some testing with Neil's patch and it causes some problems in
that if there is a hang one side of the rsync connection can hang forever.
I have instead changed it to multiplying the io_timeout by five and that
seems to help.
- Dave Dykstra
Problems begins around line 2086
1 16:30:16.312039 eth0 < dynamic.ih.lucent.com.39406 > static.8664: S
2257701238:2257701238(0) win 8760 <mss 1460> (DF)
2 16:30:16.312152 eth0 > static.8664 > dynamic.ih.lucent.com.39406: S
2711484806:2711484806(0) ack 2257701239 win 32120 <mss 1460> (DF)
3 16:30:16.312812 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 1:1(0) ack
1 win 8760 (DF)
4 16:30:16.313036 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 1:13(12)
ack 1 win 8760 (DF)
5 16:30:16.313089 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 1:1(0) ack
13 win 32120 (DF)
6 16:30:16.314922 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P 1:13(12)
ack 13 win 32120 (DF)
7 16:30:16.315713 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 13:13(0)
ack 13 win 8760 (DF)
8 16:30:16.315999 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 13:20(7)
ack 13 win 8760 (DF)
9 16:30:16.318772 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 13:13(0)
ack 20 win 32120 (DF)
10 16:30:16.326462 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P 13:25(12)
ack 20 win 32120 (DF)
11 16:30:16.327476 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 20:29(9)
ack 25 win 8760 (DF)
12 16:30:16.328759 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 25:25(0)
ack 29 win 32120 (DF)
13 16:30:16.329448 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 29:60(31)
ack 25 win 8760 (DF)
14 16:30:16.330773 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P 25:29(4)
ack 60 win 32120 (DF)
15 16:30:16.331486 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 60:64(4)
ack 29 win 8760 (DF)
16 16:30:16.338768 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 29:29(0)
ack 64 win 32120 (DF)
17 16:30:16.343753 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
29:1489(1460) ack 64 win 32120 (DF)
18 16:30:16.343804 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1489:2949(1460) ack 64 win 32120 (DF)
19 16:30:16.343837 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
2949:4125(1176) ack 64 win 32120 (DF)
20 16:30:16.349113 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 1489 win 8760 (DF)
21 16:30:16.354061 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
4125:5585(1460) ack 64 win 32120 (DF)
22 16:30:16.354101 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
5585:7045(1460) ack 64 win 32120 (DF)
23 16:30:16.354131 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
7045:8221(1176) ack 64 win 32120 (DF)
24 16:30:16.365106 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
8221:9681(1460) ack 64 win 32120 (DF)
25 16:30:16.369729 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 5585 win 8760 (DF)
26 16:30:16.369815 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
9681:11141(1460) ack 64 win 32120 (DF)
27 16:30:16.369833 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
11141:12317(1176) ack 64 win 32120 (DF)
28 16:30:16.375488 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
12317:13777(1460) ack 64 win 32120 (DF)
29 16:30:16.389681 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 11141 win 8760 (DF)
30 16:30:16.389772 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
13777:15237(1460) ack 64 win 32120 (DF)
31 16:30:16.389791 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
15237:16697(1460) ack 64 win 32120 (DF)
32 16:30:16.389806 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
16697:18157(1460) ack 64 win 32120 (DF)
33 16:30:16.389822 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
18157:19617(1460) ack 64 win 32120 (DF)
34 16:30:16.409655 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 16697 win 8760 (DF)
35 16:30:16.409741 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
19617:21077(1460) ack 64 win 32120 (DF)
36 16:30:16.409759 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
21077:22537(1460) ack 64 win 32120 (DF)
37 16:30:16.409777 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
22537:23997(1460) ack 64 win 32120 (DF)
38 16:30:16.410741 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
23997:25457(1460) ack 64 win 32120 (DF)
39 16:30:16.429704 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 21077 win 4380 (DF)
40 16:30:16.429761 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 25457 win 0 (DF)
41 16:30:16.544874 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 25457 win 8760 (DF)
42 16:30:16.544962 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
25457:26917(1460) ack 64 win 32120 (DF)
43 16:30:16.544985 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
26917:28377(1460) ack 64 win 32120 (DF)
44 16:30:16.545003 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
28377:29837(1460) ack 64 win 32120 (DF)
45 16:30:16.545021 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
29837:31297(1460) ack 64 win 32120 (DF)
46 16:30:16.545038 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
31297:32757(1460) ack 64 win 32120 (DF)
47 16:30:16.545056 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
32757:34217(1460) ack 64 win 32120 (DF)
48 16:30:16.579692 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 29837 win 8760 (DF)
49 16:30:16.579797 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
34217:35677(1460) ack 64 win 32120 (DF)
50 16:30:16.579817 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
35677:37137(1460) ack 64 win 32120 (DF)
51 16:30:16.579832 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
37137:38597(1460) ack 64 win 32120 (DF)
52 16:30:16.579753 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 34217 win 8760 (DF)
53 16:30:16.579876 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
38597:40057(1460) ack 64 win 32120 (DF)
54 16:30:16.579893 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
40057:41517(1460) ack 64 win 32120 (DF)
55 16:30:16.579909 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
41517:42977(1460) ack 64 win 32120 (DF)
56 16:30:16.599739 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 38597 win 8760 (DF)
57 16:30:16.599830 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
42977:44437(1460) ack 64 win 32120 (DF)
58 16:30:16.599850 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
44437:45897(1460) ack 64 win 32120 (DF)
59 16:30:16.599867 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
45897:47357(1460) ack 64 win 32120 (DF)
60 16:30:16.599801 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 42977 win 7300 (DF)
61 16:30:16.599912 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
47357:48817(1460) ack 64 win 32120 (DF)
62 16:30:16.599929 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
48817:50277(1460) ack 64 win 32120 (DF)
63 16:30:16.619684 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 47357 win 2920 (DF)
64 16:30:16.669642 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 50277 win 0 (DF)
65 16:30:16.745768 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 50277 win 8760 (DF)
66 16:30:16.745851 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
50277:51737(1460) ack 64 win 32120 (DF)
67 16:30:16.745873 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
51737:53197(1460) ack 64 win 32120 (DF)
68 16:30:16.745892 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
53197:54657(1460) ack 64 win 32120 (DF)
69 16:30:16.745908 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
54657:56117(1460) ack 64 win 32120 (DF)
70 16:30:16.745923 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
56117:57577(1460) ack 64 win 32120 (DF)
71 16:30:16.745938 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
57577:58755(1178) ack 64 win 32120 (DF)
72 16:30:16.766697 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 53197 win 8760 (DF)
73 16:30:16.789642 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 56117 win 8760 (DF)
74 16:30:16.839614 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 64:64(0)
ack 58755 win 8760 (DF)
75 16:30:17.553776 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
64:1524(1460) ack 58755 win 8760 (DF)
76 16:30:17.555005 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P
1524:2984(1460) ack 58755 win 8760 (DF)
77 16:30:17.555078 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
58755:58755(0) ack 2984 win 30660 (DF)
78 16:30:17.556008 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P
2984:4156(1172) ack 58755 win 8760 (DF)
79 16:30:17.602616 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
58755:60215(1460) ack 4156 win 29488 (DF)
80 16:30:17.602687 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
60215:61675(1460) ack 4156 win 29488 (DF)
81 16:30:17.602723 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
61675:62851(1176) ack 4156 win 29488 (DF)
82 16:30:17.603531 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
62851:64311(1460) ack 4156 win 29488 (DF)
83 16:30:17.603571 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
64311:65771(1460) ack 4156 win 29488 (DF)
84 16:30:17.603607 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
65771:66947(1176) ack 4156 win 29488 (DF)
85 16:30:17.619674 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
4156:4156(0) ack 61675 win 8760 (DF)
86 16:30:17.619828 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
66947:68370(1423) ack 4156 win 29488 (DF)
87 16:30:17.619726 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
4156:4156(0) ack 65771 win 8760 (DF)
88 16:30:17.639631 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
4156:4156(0) ack 68370 win 8760 (DF)
89 16:30:17.647401 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
68370:69830(1460) ack 4156 win 29488 (DF)
90 16:30:17.647444 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
69830:71290(1460) ack 4156 win 29488 (DF)
91 16:30:17.647481 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
71290:72466(1176) ack 4156 win 29488 (DF)
92 16:30:17.669655 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
4156:4156(0) ack 71290 win 8760 (DF)
93 16:30:17.719581 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
4156:4156(0) ack 72466 win 8760 (DF)
94 16:30:17.719680 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
72466:73049(583) ack 4156 win 29488 (DF)
95 16:30:17.724487 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
73049:74509(1460) ack 4156 win 29488 (DF)
96 16:30:17.724531 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
74509:75969(1460) ack 4156 win 29488 (DF)
97 16:30:17.724564 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
75969:77145(1176) ack 4156 win 29488 (DF)
98 16:30:17.725133 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
77145:78605(1460) ack 4156 win 29488 (DF)
99 16:30:17.725172 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
78605:80065(1460) ack 4156 win 29488 (DF)
100 16:30:17.739678 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
4156:4156(0) ack 75969 win 8760 (DF)
...
2000 16:31:41.489169 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1533148:1534608(1460) ack 67580 win 0 (DF)
2001 16:31:41.489234 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1534608:1536068(1460) ack 67580 win 0 (DF)
2002 16:31:41.489251 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1536068:1537001(933) ack 67580 win 0 (DF)
2003 16:31:41.489202 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1533148 win 7300 (DF)
2004 16:31:41.493177 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1537001:1538461(1460) ack 67580 win 0 (DF)
2005 16:31:41.509187 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1536068 win 4380 (DF)
2006 16:31:41.536938 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1538461:1539921(1460) ack 67580 win 0 (DF)
2007 16:31:41.559218 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1539921 win 527 (DF)
2008 16:31:41.758796 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1539921:1540448(527) ack 67580 win 0 (DF)
2009 16:31:41.829153 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1540448 win 0 (DF)
2010 16:31:42.013036 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1540448 win 8760 (DF)
2011 16:31:42.013088 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1540448:1541381(933) ack 67580 win 0 (DF)
2012 16:31:42.013109 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1541381:1542841(1460) ack 67580 win 0 (DF)
2013 16:31:42.013127 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1542841:1544301(1460) ack 67580 win 0 (DF)
2014 16:31:42.013144 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1544301:1545761(1460) ack 67580 win 0 (DF)
2015 16:31:42.013161 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1545761:1547221(1460) ack 67580 win 0 (DF)
2016 16:31:42.013178 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1547221:1548681(1460) ack 67580 win 0 (DF)
2017 16:31:42.021930 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1544301 win 8760 (DF)
2018 16:31:42.022031 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1548681:1550141(1460) ack 67580 win 0 (DF)
2019 16:31:42.022050 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1550141:1551601(1460) ack 67580 win 0 (DF)
2020 16:31:42.022066 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1551601:1553061(1460) ack 67580 win 0 (DF)
2021 16:31:42.021991 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1547221 win 8760 (DF)
2022 16:31:42.022105 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1553061:1554521(1460) ack 67580 win 0 (DF)
2023 16:31:42.022122 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1554521:1555981(1460) ack 67580 win 0 (DF)
2024 16:31:42.039230 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1550141 win 8760 (DF)
2025 16:31:42.039334 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1555981:1557441(1460) ack 67580 win 0 (DF)
2026 16:31:42.039385 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1557441:1558901(1460) ack 67580 win 0 (DF)
2027 16:31:42.039291 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1553061 win 8760 (DF)
2028 16:31:42.039426 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1558901:1560361(1460) ack 67580 win 0 (DF)
2029 16:31:42.039442 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1560361:1561821(1460) ack 67580 win 0 (DF)
2030 16:31:42.039366 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1555981 win 8760 (DF)
2031 16:31:42.039477 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1561821:1563281(1460) ack 67580 win 0 (DF)
2032 16:31:42.039493 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1563281:1564741(1460) ack 67580 win 0 (DF)
2033 16:31:42.059242 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1558901 win 7300 (DF)
2034 16:31:42.059359 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1564741:1566201(1460) ack 67580 win 0 (DF)
2035 16:31:42.059305 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1561821 win 4380 (DF)
2036 16:31:42.059377 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1564741 win 1460 (DF)
2037 16:31:42.129158 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1566201 win 0 (DF)
2038 16:31:42.328765 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1566200:1566200(0) ack 67580 win 0 (DF)
2039 16:31:42.329606 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1566201 win 0 (DF)
2040 16:31:42.728757 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1566200:1566200(0) ack 67580 win 0 (DF)
2041 16:31:42.729551 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1566201 win 0 (DF)
2042 16:31:42.844554 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1566201 win 8760 (DF)
2043 16:31:42.844609 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1566201:1567661(1460) ack 67580 win 0 (DF)
2044 16:31:42.844629 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1567661:1569121(1460) ack 67580 win 0 (DF)
2045 16:31:42.844647 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1569121:1570581(1460) ack 67580 win 0 (DF)
2046 16:31:42.844665 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1570581:1572041(1460) ack 67580 win 0 (DF)
2047 16:31:42.844682 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1572041:1573501(1460) ack 67580 win 0 (DF)
2048 16:31:42.844701 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1573501:1574961(1460) ack 67580 win 0 (DF)
2049 16:31:42.859216 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1567661 win 8760 (DF)
2050 16:31:42.859314 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1574961:1576421(1460) ack 67580 win 0 (DF)
2051 16:31:42.859280 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1570581 win 8760 (DF)
2052 16:31:42.859386 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1576421:1577881(1460) ack 67580 win 0 (DF)
2053 16:31:42.859403 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1577881:1579341(1460) ack 67580 win 0 (DF)
2054 16:31:42.859354 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1573501 win 8760 (DF)
2055 16:31:42.859438 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1579341:1580801(1460) ack 67580 win 0 (DF)
2056 16:31:42.859455 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1580801:1582261(1460) ack 67580 win 0 (DF)
2057 16:31:42.879205 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1576421 win 7300 (DF)
2058 16:31:42.879315 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1582261:1583721(1460) ack 67580 win 0 (DF)
2059 16:31:42.879268 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1579341 win 4380 (DF)
2060 16:31:42.879337 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67580(0) ack 1582261 win 1460 (DF)
2061 16:31:42.907940 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 67580 win 1460 (DF)
2062 16:31:42.908620 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
67580:67581(1) ack 1583721 win 0 (DF)
2063 16:31:43.098761 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 67581 win 1460 (DF)
2064 16:31:43.100993 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P
67581:68456(875) ack 1583721 win 0 (DF)
2065 16:31:43.101524 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P
68456:69041(585) ack 1583721 win 0 (DF)
2066 16:31:43.108759 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583720:1583720(0) ack 69041 win 0 (DF)
2067 16:31:43.110623 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P
69041:69628(587) ack 1583721 win 0 (DF)
2068 16:31:43.110679 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 69041 win 0 (DF)
2069 16:31:43.111641 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P
69628:70501(873) ack 1583721 win 0 (DF)
2070 16:31:43.111716 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 69041 win 0 (DF)
2071 16:31:43.111707 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1583721 win 0 (DF)
2072 16:31:43.111748 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 69041 win 0 (DF)
2073 16:31:43.508760 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583720:1583720(0) ack 69041 win 0 (DF)
2074 16:31:43.509575 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1583721 win 0 (DF)
2075 16:31:43.509629 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 69041 win 0 (DF)
2076 16:31:43.521755 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1583721 win 0 (DF)
2077 16:31:43.521840 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1583721(0) ack 69041 win 0 (DF)
2078 16:31:43.896657 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:69041(0) ack 1583721 win 8760 (DF)
2079 16:31:43.896717 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2080 16:31:43.896735 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1585181:1586641(1460) ack 69041 win 0 (DF)
2081 16:31:43.896754 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1586641:1588101(1460) ack 69041 win 0 (DF)
2082 16:31:43.896771 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1588101:1589561(1460) ack 69041 win 0 (DF)
2083 16:31:43.896786 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1589561:1591021(1460) ack 69041 win 0 (DF)
2084 16:31:43.896804 eth0 > static.8664 > dynamic.ih.lucent.com.39406: P
1591021:1592481(1460) ack 69041 win 0 (DF)
2085 16:31:43.904440 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1585181 win 8760 (DF)
2086 16:31:43.904489 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2087 16:31:43.906506 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1588101 win 8760 (DF)
2088 16:31:43.906548 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2089 16:31:43.906570 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1591021 win 8760 (DF)
2090 16:31:43.906603 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2091 16:31:43.949156 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2092 16:31:43.949203 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2093 16:31:44.088759 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2094 16:31:44.092205 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2095 16:31:44.092251 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2096 16:31:44.341765 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2097 16:31:44.341829 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2098 16:31:44.488774 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2099 16:31:44.492160 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2100 16:31:44.492207 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2101 16:31:45.288771 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2102 16:31:45.292128 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2103 16:31:45.292177 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2104 16:31:45.981721 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2105 16:31:45.981790 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2106 16:31:46.888758 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2107 16:31:46.892128 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2108 16:31:46.892176 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2109 16:31:49.271708 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2110 16:31:49.271776 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2111 16:31:50.088776 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2112 16:31:50.092118 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2113 16:31:50.092167 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2114 16:31:55.861786 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2115 16:31:55.861863 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2116 16:31:56.488774 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2117 16:31:56.492191 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2118 16:31:56.492236 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2119 16:32:09.041585 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2120 16:32:09.041659 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2121 16:32:09.288759 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2122 16:32:09.292213 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2123 16:32:09.292261 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2124 16:32:34.888769 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2125 16:32:34.892162 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2126 16:32:34.892215 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2127 16:32:35.401430 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2128 16:32:35.401491 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2129 16:33:26.088780 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2130 16:33:26.092161 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2131 16:33:26.092218 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2132 16:33:28.131002 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2133 16:33:28.131074 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2134 16:34:28.130511 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2135 16:34:28.130598 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2136 16:35:08.488774 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2137 16:35:08.492158 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2138 16:35:08.492214 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2139 16:35:28.130065 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2140 16:35:28.130151 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2141 16:36:28.129534 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2142 16:36:28.129625 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2143 16:37:08.488774 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1583721:1585181(1460) ack 69041 win 0 (DF)
2144 16:37:08.492204 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
70501:70501(0) ack 1592481 win 8760 (DF)
2145 16:37:08.492259 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
2146 16:37:28.129120 eth0 < dynamic.ih.lucent.com.39406 > static.8664: .
69041:70501(1460) ack 1592481 win 8760 (DF)
2147 16:37:28.129205 eth0 > static.8664 > dynamic.ih.lucent.com.39406: .
1592481:1592481(0) ack 69041 win 0 (DF)
