Re: http://lists.samba.org/pipermail/rsync/2001-February/003628.html
Some more email was exhanged the last couple days on the subject of a TCP
hang between Solaris and Linux:
------------------------------------------------------------------------
> Date: Mon, 26 Feb 2001 12:06:08 -0600
> From: Dave Dykstra <[EMAIL PROTECTED]>
> To: "David S. Miller" <[EMAIL PROTECTED]>
> Cc: [EMAIL PROTECTED], [EMAIL PROTECTED]
> Subject: Re: Linux 2.2.16 through 2.2.18preX TCP hang bug triggered by rsync
>
> David,
>
> Regarding the patch you sent us (below) to try to help solve the problem of
> Solaris sending acks with a sequence number that was out-of-window:
>
> We have now completed extensive testing with it and analysis of tcpdumps
> and have determined that the patch is working as expected (accepting the
> acks) but it isn't enough to work around the state that Solaris gets itself
> into; the connection still hangs. It looks like Alexey was right. Linux
> is able to make further progress getting data sent to Solaris but it isn't
> enough to recover the whole session; the Linux receive window stays at 0 so
> I presume the rsync application isn't reading the data because it's waiting
> for the Solaris side to complete something. Oddly, every 30 or 60 seconds
> after this situation occurs, Linux sends another 2-4 1460 byte packets and
> they're acknowledged by Solaris. It seems unlikely that the rsync
> application would be sending exact multiples of 1460, but I didn't do a
> trace during the hang to see if it was generating extra data for some
> reason.
>
> I have attached the tcpdump in case you're interested. Recall that 'static'
> is Linux and 'dynamic' is Solaris. We have added our interpretation on
> some of the lines.
>
> We also have had an initial response from Sun where they recommended
> upgrading with a certain patch but that too hasn't solved the problem (the
> attached tcpdump is with the Solaris patch in place).
>
> Thanks for your help and I'll let you know if we do ever get a satisfactory
> answer from Sun.
>
> - Dave Dykstra
------------------------------------------------------------------------
> Date: Tue, 27 Feb 2001 13:13:32 -0600
> From: Dave Dykstra <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED],
> [EMAIL PROTECTED]
> Subject: Re: Linux 2.2.16 through 2.2.18preX TCP hang bug triggered by rsync
>
>
> On Tue, Feb 27, 2001 at 09:51:36PM +0300, [EMAIL PROTECTED] wrote:
> > Hello!
> >
> > > into; the connection still hangs. It looks like Alexey was right. Linux
> > > is able to make further progress getting data sent to Solaris but it isn't
> > > enough to recover the whole session; the Linux receive window stays at 0 so
> > > I presume the rsync application isn't reading the data because it's waiting
> > > for the Solaris side to complete something. Oddly, every 30 or 60 seconds
> > > after this situation occurs, Linux sends another 2-4 1460 byte packets and
> > > they're acknowledged by Solaris.
> >
> > The situation is rather opposite. Solaris does not enter persist mode (#1)
> > and sets SND.NXT to an invalid value (#2) (these are two different problems).
> > So, its ACKs are valid only when it retransmits. With changes made
> > at Linux side the session does not hang, but becomes clocked by Solaris
> > retransmission timer. When Solaris retransmits, we see ACK and send
> > to new window. Apparently, if window from linux side is not open
> > for tcp timeout, solaris aborts session.
>
> That makes sense.
>
>
> > I talked to sun engineer; he said they are aware of both these bugs,
> > but patch fixing this is still not available.
>
> I'm glad to hear that it's gotten to the engineer who will be able to make
> a difference. Could you please give me his email address? When a patch is
> available I'm willing to test it for him.
>
>
> > [ Also, he was suprised with rsync. 8)8) Though transmissions with closed
> > window are legal and it would work if solaris was not buggy,
> > I have to agree: this application is really stupid, it uses tcp
> > in maximally suboptimal way.
> > ]
> >
> > Actually, this place is one the most thin place in tcp protocol.
> > Due to obscurity of specifications, most of OSes have bugs here
> > (both fatal like solaris and not fatal, but still affecting performance).
> > To avoid such problems maintainers of rsync could take care of not holding
> > window closed.
>
> I have added the rsync author to the Cc and will forward your message to
> the rsync mailing list.
>
>
> > Moreover, I have seen report on deadlock in rsync itself,
> > when both sides stay with zero window forever, because both sides
> > want to write.
>
> I have been closely following rsync for a couple years now and have never
> seen a confirmed case of that, so I'm quite skeptical. There have been
> numerous cases of triggering OS bugs, however, so I'm not surprised by your
> comment about it being the "most thin place" in the tcp protocol.
>
> Thanks!
>
> - Dave Dykstra
diff -u --recursive --new-file --exclude=CVS --exclude=.cvsignore
../vanilla/2.2/linux/net/ipv4/tcp_input.c linux/net/ipv4/tcp_input.c
--- ../vanilla/2.2/linux/net/ipv4/tcp_input.c Sun Dec 10 16:49:44 2000
+++ linux/net/ipv4/tcp_input.c Thu Feb 1 17:50:39 2001
@@ -5,7 +5,7 @@
*
* Implementation of the Transmission Control Protocol(TCP).
*
- * Version: $Id: tcp_input.c,v 1.164.2.18 2000/12/08 20:29:33 davem Exp $
+ * Version: $Id: tcp_input.c,v 1.164.2.19 2001/02/02 01:50:39 davem Exp $
*
* Authors: Ross Biro, <[EMAIL PROTECTED]>
* Fred N. van Kempen, <[EMAIL PROTECTED]>
@@ -246,26 +246,22 @@
}
-static int __tcp_sequence(struct tcp_opt *tp, u32 seq, u32 end_seq)
-{
- u32 end_window = tp->rcv_wup + tp->rcv_wnd;
-
- if (tp->rcv_wnd &&
- after(end_seq, tp->rcv_nxt) &&
- before(seq, end_window))
- return 1;
- if (seq != end_window)
- return 0;
- return (seq == end_seq);
-}
-
-/* This functions checks to see if the tcp header is actually acceptable. */
-extern __inline__ int tcp_sequence(struct tcp_opt *tp, u32 seq, u32 end_seq)
+/* Check segment sequence number for validity.
+ *
+ * Segment controls are considered valid, if the segment
+ * fits to the window after truncation to the window. Acceptability
+ * of data (and SYN, FIN, of course) is checked separately.
+ * See tcp_data_queue(), for example.
+ *
+ * Also, controls (RST is main one) are accepted using RCV.WUP instead
+ * of RCV.NXT. Peer still did not advance his SND.UNA when we
+ * delayed ACK, so that hisSND.UNA<=ourRCV.WUP.
+ * (borrowed from freebsd)
+ */
+static inline int tcp_sequence(struct tcp_opt *tp, u32 seq, u32 end_seq)
{
- if (seq == tp->rcv_nxt)
- return (tp->rcv_wnd || (end_seq == seq));
-
- return __tcp_sequence(tp, seq, end_seq);
+ return !before(end_seq, tp->rcv_wup) &&
+ !after(seq, tp->rcv_nxt + tcp_receive_window(tp));
}
/* When we get a reset we do this. */
@@ -1415,6 +1411,11 @@
if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
/* Ok. In sequence. */
queue_and_out:
+ if (tcp_receive_window(tp) == 0)
+ goto out_of_window;
+
+ /* We know it is in window now too. */
+
dst_confirm(sk->dst_cache);
__skb_queue_tail(&sk->receive_queue, skb);
tp->rcv_nxt = TCP_SKB_CB(skb)->end_seq;
@@ -1441,6 +1442,7 @@
/* A retransmit, 2nd most common case. Force an imediate ack. */
SOCK_DEBUG(sk, "retransmit received: seq %X\n", TCP_SKB_CB(skb)->seq);
tcp_enter_quickack_mode(tp);
+out_of_window:
kfree_skb(skb);
return;
}
@@ -1453,6 +1455,16 @@
goto queue_and_out;
}
+
+ /* Out of window. F.e. zero window probe.
+ *
+ * Note: it is highly possible that we may open window and enqueue
+ * this segment now. However, this will be known only after we queue
+ * it, which will result in queue full of successive 1 byte BSD
+ * window probes, it is SWS in fact. So, always reject it and send ACK.
+ */
+ if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt+tcp_receive_window(tp)))
+ goto out_of_window;
/* Ok. This is an out_of_order segment, force an ack. */
tp->delayed_acks++;
18:53:57.739331 eth0 < dynamic.32921 > static.8664: . 67294:67294(0) ack 1639367 win
8760 (DF)
18:53:57.739418 eth0 > static.8664 > dynamic.32921: . 1645207:1646667(1460) ack 67294
win 0 (DF)
18:53:57.739438 eth0 > static.8664 > dynamic.32921: P 1646667:1648127(1460) ack 67294
win 0 (DF)
18:53:57.747477 eth0 > static.8664 > dynamic.32921: . 1648127:1648127(0) ack 67294 win
1460 (DF)
18:53:57.748256 eth0 < dynamic.32921 > static.8664: . 67294:67294(0) ack 1642287 win
5840 (DF)
18:53:57.748323 eth0 < dynamic.32921 > static.8664: . 67294:67294(0) ack 1645207 win
2920 (DF)
18:53:57.748395 eth0 < dynamic.32921 > static.8664: . 67294:67294(0) ack 1648127 win 0
(DF)
18:53:57.748464 eth0 < dynamic.32921 > static.8664: . 67294:67295(1) ack 1648127 win 0
(DF)
18:53:58.008822 eth0 > static.8664 > dynamic.32921: . 1648127:1648127(0) ack 67295 win
1460 (DF) Static acks 67295 win=1460
18:53:58.011149 eth0 < dynamic.32921 > static.8664: P 67295:68464(1169) ack 1648127
win 0 (DF) Dynamic sends 68464
18:53:58.011444 eth0 < dynamic.32921 > static.8664: P 68464:68755(291) ack 1648127 win
0 (DF) Dynamic sends 68755, filling window
18:53:58.018813 eth0 > static.8664 > dynamic.32921: . 1648126:1648126(0) ack 68755 win
0 (DF) static acks 68755, but win=0
18:53:58.020724 eth0 < dynamic.32921 > static.8664: P 68755:69636(881) ack 1648127 win
0 (DF) Dynamic send outside of window
18:53:58.021248 eth0 < dynamic.32921 > static.8664: P 69636:70215(579) ack 1648127 win
0 (DF) Dynamic sends 70215, outside of window
18:53:58.021295 eth0 > static.8664 > dynamic.32921: . 1648127:1648127(0) ack 68755 win
0 (DF) Static acks 68755
18:53:58.021315 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1648127 win 0
(DF) Dynamic acks 1648127
18:53:58.021354 eth0 > static.8664 > dynamic.32921: . 1648127:1648127(0) ack 68755 win
0 (DF) Static acks 68755 win=0
18:53:58.511556 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1648127
win 0 (DF) Dynamic resends 68755:70215, but that's outside the window
18:53:58.878252 eth0 < dynamic.32921 > static.8664: . 68755:68755(0) ack 1648127 win
8760 (DF)
18:53:58.878326 eth0 > static.8664 > dynamic.32921: . 1648127:1649587(1460) ack 68755
win 0 (DF)
18:53:58.878349 eth0 > static.8664 > dynamic.32921: . 1649587:1651047(1460) ack 68755
win 0 (DF)
18:53:58.878369 eth0 > static.8664 > dynamic.32921: P 1651047:1652507(1460) ack 68755
win 0 (DF)
18:53:58.878389 eth0 > static.8664 > dynamic.32921: P 1652507:1653967(1460) ack 68755
win 0 (DF)
18:53:58.878408 eth0 > static.8664 > dynamic.32921: P 1653967:1655427(1460) ack 68755
win 0 (DF)
18:53:58.878428 eth0 > static.8664 > dynamic.32921: P 1655427:1656887(1460) ack 68755
win 0 (DF)
18:53:58.899155 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1649587 win
8760 (DF)
18:53:58.899226 eth0 > static.8664 > dynamic.32921: . 1656887:1656887(0) ack 68755 win
0 (DF)
18:53:58.899218 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1652507 win
8760 (DF)
18:53:58.899263 eth0 > static.8664 > dynamic.32921: . 1656887:1656887(0) ack 68755 win
0 (DF)
18:53:58.899293 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1655427 win
8760 (DF)
18:53:58.899322 eth0 > static.8664 > dynamic.32921: . 1656887:1656887(0) ack 68755 win
0 (DF)
18:53:58.999139 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1656887 win
8760 (DF)
18:53:58.999187 eth0 > static.8664 > dynamic.32921: . 1656887:1656887(0) ack 68755 win
0 (DF)
18:53:59.138821 eth0 > static.8664 > dynamic.32921: . 1648127:1649587(1460) ack 68755
win 0 (DF)
18:53:59.141677 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1656887 win
8760 (DF)
18:53:59.141728 eth0 > static.8664 > dynamic.32921: . 1656887:1656887(0) ack 68755 win
0 (DF)
18:53:59.491549 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1656887
win 8760 (DF)
18:53:59.491667 eth0 > static.8664 > dynamic.32921: . 1656887:1658347(1460) ack 68755
win 0 (DF)
18:53:59.491690 eth0 > static.8664 > dynamic.32921: . 1658347:1659807(1460) ack 68755
win 0 (DF)
18:53:59.495603 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1658347 win
8760 (DF)
18:53:59.495651 eth0 > static.8664 > dynamic.32921: . 1659807:1659807(0) ack 68755 win
0 (DF)
18:53:59.589158 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1659807 win
8760 (DF)
18:53:59.589211 eth0 > static.8664 > dynamic.32921: . 1659807:1659807(0) ack 68755 win
0 (DF)
18:54:00.028844 eth0 > static.8664 > dynamic.32921: . 1656887:1658347(1460) ack 68755
win 0 (DF)
18:54:00.031722 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1659807 win
8760 (DF)
18:54:00.031776 eth0 > static.8664 > dynamic.32921: . 1659807:1659807(0) ack 68755 win
0 (DF)
18:54:01.108840 eth0 > static.8664 > dynamic.32921: . 1656887:1658347(1460) ack 68755
win 0 (DF)
18:54:01.111746 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1659807 win
8760 (DF)
18:54:01.111812 eth0 > static.8664 > dynamic.32921: . 1659807:1659807(0) ack 68755 win
0 (DF)
18:54:01.451563 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1659807
win 8760 (DF)
18:54:01.451689 eth0 > static.8664 > dynamic.32921: P 1659807:1661267(1460) ack 68755
win 0 (DF)
18:54:01.451713 eth0 > static.8664 > dynamic.32921: . 1661267:1662727(1460) ack 68755
win 0 (DF)
18:54:01.455627 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1661267 win
8760 (DF)
18:54:01.455668 eth0 > static.8664 > dynamic.32921: . 1662727:1662727(0) ack 68755 win
0 (DF)
18:54:01.549122 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1662727 win
8760 (DF)
18:54:01.549173 eth0 > static.8664 > dynamic.32921: . 1662727:1662727(0) ack 68755 win
0 (DF)
18:54:03.608849 eth0 > static.8664 > dynamic.32921: P 1659807:1661267(1460) ack 68755
win 0 (DF)
18:54:03.611755 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1662727 win
8760 (DF)
18:54:03.611813 eth0 > static.8664 > dynamic.32921: . 1662727:1662727(0) ack 68755 win
0 (DF)
18:54:05.371648 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1662727
win 8760 (DF)
18:54:05.371768 eth0 > static.8664 > dynamic.32921: P 1662727:1664187(1460) ack 68755
win 0 (DF)
18:54:05.371793 eth0 > static.8664 > dynamic.32921: P 1664187:1665647(1460) ack 68755
win 0 (DF)
18:54:05.375712 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1664187 win
8760 (DF)
18:54:05.375759 eth0 > static.8664 > dynamic.32921: . 1665647:1665647(0) ack 68755 win
0 (DF)
18:54:05.469122 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1665647 win
8760 (DF)
18:54:05.469196 eth0 > static.8664 > dynamic.32921: . 1665647:1665647(0) ack 68755 win
0 (DF)
18:54:09.688833 eth0 > static.8664 > dynamic.32921: P 1662727:1664187(1460) ack 68755
win 0 (DF)
18:54:09.691753 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1665647 win
8760 (DF)
18:54:09.691813 eth0 > static.8664 > dynamic.32921: . 1665647:1665647(0) ack 68755 win
0 (DF)
18:54:13.219371 eth0 B arp who-has static (Broadcast) tell dynamic
18:54:13.219419 eth0 > arp reply static (0:60:b0:ee:6:2e) is-at 0:60:b0:ee:6:2e
(8:0:20:a6:bc:8)
18:54:13.222303 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1665647
win 8760 (DF)
18:54:13.222424 eth0 > static.8664 > dynamic.32921: P 1665647:1667107(1460) ack 68755
win 0 (DF)
18:54:13.222446 eth0 > static.8664 > dynamic.32921: P 1667107:1668567(1460) ack 68755
win 0 (DF)
18:54:13.226406 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1667107 win
8760 (DF)
18:54:13.226448 eth0 > static.8664 > dynamic.32921: . 1668567:1668567(0) ack 68755 win
0 (DF)
18:54:13.319106 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1668567 win
8760 (DF)
18:54:13.319152 eth0 > static.8664 > dynamic.32921: . 1668567:1668567(0) ack 68755 win
0 (DF)
18:54:21.858851 eth0 > static.8664 > dynamic.32921: P 1665647:1667107(1460) ack 68755
win 0 (DF)
18:54:21.861820 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1668567 win
8760 (DF)
18:54:21.861881 eth0 > static.8664 > dynamic.32921: . 1668567:1668567(0) ack 68755 win
0 (DF)
18:54:28.931507 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1668567
win 8760 (DF)
18:54:28.931632 eth0 > static.8664 > dynamic.32921: P 1668567:1670027(1460) ack 68755
win 0 (DF)
18:54:28.931657 eth0 > static.8664 > dynamic.32921: P 1670027:1671487(1460) ack 68755
win 0 (DF)
18:54:28.935575 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1670027 win
8760 (DF)
18:54:28.935621 eth0 > static.8664 > dynamic.32921: . 1671487:1671487(0) ack 68755 win
0 (DF)
18:54:29.029093 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1671487 win
8760 (DF)
18:54:29.029142 eth0 > static.8664 > dynamic.32921: . 1671487:1671487(0) ack 68755 win
0 (DF)
18:54:46.208834 eth0 > static.8664 > dynamic.32921: P 1668567:1670027(1460) ack 68755
win 0 (DF)
18:54:46.211803 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1671487 win
8760 (DF)
18:54:46.211864 eth0 > static.8664 > dynamic.32921: . 1671487:1671487(0) ack 68755 win
0 (DF)
18:55:00.351430 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1671487
win 8760 (DF)
18:55:00.351549 eth0 > static.8664 > dynamic.32921: P 1671487:1672947(1460) ack 68755
win 0 (DF)
18:55:00.351574 eth0 > static.8664 > dynamic.32921: P 1672947:1674407(1460) ack 68755
win 0 (DF)
18:55:00.355486 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1672947 win
8760 (DF)
18:55:00.355534 eth0 > static.8664 > dynamic.32921: . 1674407:1674407(0) ack 68755 win
0 (DF)
18:55:00.448980 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1674407 win
8760 (DF)
18:55:00.449033 eth0 > static.8664 > dynamic.32921: . 1674407:1674407(0) ack 68755 win
0 (DF)
18:55:34.908842 eth0 > static.8664 > dynamic.32921: P 1671487:1672947(1460) ack 68755
win 0 (DF)
18:55:34.911810 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1674407 win
8760 (DF)
18:55:34.911872 eth0 > static.8664 > dynamic.32921: . 1674407:1674407(0) ack 68755 win
0 (DF)
18:56:00.351256 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1674407
win 8760 (DF)
18:56:00.351396 eth0 > static.8664 > dynamic.32921: P 1674407:1675867(1460) ack 68755
win 0 (DF)
18:56:00.351423 eth0 > static.8664 > dynamic.32921: P 1675867:1677327(1460) ack 68755
win 0 (DF)
18:56:00.355334 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1675867 win
8760 (DF)
18:56:00.355379 eth0 > static.8664 > dynamic.32921: . 1677327:1677327(0) ack 68755 win
0 (DF)
18:56:00.448813 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1677327 win
8760 (DF)
18:56:00.448868 eth0 > static.8664 > dynamic.32921: . 1677327:1677327(0) ack 68755 win
0 (DF)
18:57:00.351088 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1677327
win 8760 (DF)
18:57:00.351225 eth0 > static.8664 > dynamic.32921: P 1677327:1678787(1460) ack 68755
win 0 (DF)
18:57:00.351251 eth0 > static.8664 > dynamic.32921: P 1678787:1680247(1460) ack 68755
win 0 (DF)
18:57:00.351272 eth0 > static.8664 > dynamic.32921: P 1680247:1681707(1460) ack 68755
win 0 (DF)
18:57:00.368799 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1680247 win
8760 (DF)
18:57:00.368852 eth0 > static.8664 > dynamic.32921: . 1681707:1681707(0) ack 68755 win
0 (DF)
18:57:00.468619 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1681707 win
8760 (DF)
18:57:00.468677 eth0 > static.8664 > dynamic.32921: . 1681707:1681707(0) ack 68755 win
0 (DF)
18:58:00.350921 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1681707
win 8760 (DF)
18:58:00.351058 eth0 > static.8664 > dynamic.32921: P 1681707:1683167(1460) ack 68755
win 0 (DF)
18:58:00.351085 eth0 > static.8664 > dynamic.32921: P 1683167:1684627(1460) ack 68755
win 0 (DF)
18:58:00.351105 eth0 > static.8664 > dynamic.32921: P 1684627:1686087(1460) ack 68755
win 0 (DF)
18:58:00.368619 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1684627 win
8760 (DF)
18:58:00.368675 eth0 > static.8664 > dynamic.32921: . 1686087:1686087(0) ack 68755 win
0 (DF)
18:58:00.468459 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1686087 win
8760 (DF)
18:58:00.468518 eth0 > static.8664 > dynamic.32921: . 1686087:1686087(0) ack 68755 win
0 (DF)
18:59:00.350736 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1686087
win 8760 (DF)
18:59:00.350872 eth0 > static.8664 > dynamic.32921: P 1686087:1687547(1460) ack 68755
win 0 (DF)
18:59:00.350899 eth0 > static.8664 > dynamic.32921: P 1687547:1689007(1460) ack 68755
win 0 (DF)
18:59:00.350920 eth0 > static.8664 > dynamic.32921: P 1689007:1690467(1460) ack 68755
win 0 (DF)
18:59:00.368558 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1689007 win
8760 (DF)
18:59:00.368608 eth0 > static.8664 > dynamic.32921: . 1690467:1690467(0) ack 68755 win
0 (DF)
18:59:00.468273 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1690467 win
8760 (DF)
18:59:00.468327 eth0 > static.8664 > dynamic.32921: . 1690467:1690467(0) ack 68755 win
0 (DF)
19:00:00.350594 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1690467
win 8760 (DF)
19:00:00.350761 eth0 > static.8664 > dynamic.32921: P 1690467:1691927(1460) ack 68755
win 0 (DF)
19:00:00.350789 eth0 > static.8664 > dynamic.32921: P 1691927:1693387(1460) ack 68755
win 0 (DF)
19:00:00.350812 eth0 > static.8664 > dynamic.32921: P 1693387:1694847(1460) ack 68755
win 0 (DF)
19:00:00.368271 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1693387 win
8760 (DF)
19:00:00.368322 eth0 > static.8664 > dynamic.32921: . 1694847:1694847(0) ack 68755 win
0 (DF)
19:00:00.468106 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1694847 win
8760 (DF)
19:00:00.468159 eth0 > static.8664 > dynamic.32921: . 1694847:1694847(0) ack 68755 win
0 (DF)
19:01:00.350403 eth0 < dynamic.32921 > static.8664: . 68755:70215(1460) ack 1694847
win 8760 (DF)
19:01:00.350569 eth0 > static.8664 > dynamic.32921: P 1694847:1696307(1460) ack 68755
win 0 (DF)
19:01:00.350596 eth0 > static.8664 > dynamic.32921: P 1696307:1697767(1460) ack 68755
win 0 (DF)
19:01:00.350617 eth0 > static.8664 > dynamic.32921: P 1697767:1699227(1460) ack 68755
win 0 (DF)
19:01:00.350639 eth0 > static.8664 > dynamic.32921: P 1699227:1700687(1460) ack 68755
win 0 (DF)
19:01:00.368108 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1697767 win
8760 (DF)
19:01:00.368182 eth0 > static.8664 > dynamic.32921: . 1700687:1700687(0) ack 68755 win
0 (DF)
19:01:00.368173 eth0 < dynamic.32921 > static.8664: . 70215:70215(0) ack 1700687 win
8760 (DF)
019:01:00.368222 eth0 > static.8664 > dynamic.32921: . 1700687:1700687(0) ack 68755
win 0 (DF)
19:02:05.398817 eth0 > arp who-has dynamic tell static (0:60:b0:ee:6:2e)
19:02:05.399276 eth0 < arp reply dynamic is-at 8:0:20:a6:bc:8 (0:60:b0:ee:6:2e)
19:02:17.192157 eth0 < dynamic.32927 > static.9016: S 1156788666:1156788666(0) win
8760 <mss 1460> (DF)
19:02:17.192243 eth0 > static.9016 > dynamic.32927: R 0:0(0) ack 1156788667 win 0
