TCP connections timing out real fast

[Robert Watson]

Don't yet have any quantitative evidence that this is the case, but I feel
like TCP sessions have been timing out on me a lot faster than they used
to.  For example, yesterday a machine got unplugged from the network for
about 15 seconds: in that time, the SSH sessions to the machine timed out
and disconnected.  This morning, a machine generated a lot of output to
the serial console keeping it substantially busy for about 20 seconds; in
that time, the SSH session to it timed out.  I'm going to see if I can't
generate some tcpdump traces later today to confirm my suspicions, but was
wondering if anyone else (annecdotally or not) has seen similar things? 

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: TCP connections timing out real fast

[Bosko Milekic]

On Sat, Feb 22, 2003 at 10:57:05AM -0500, Robert Watson wrote:
 
 Don't yet have any quantitative evidence that this is the case, but I feel
 like TCP sessions have been timing out on me a lot faster than they used
 to.  For example, yesterday a machine got unplugged from the network for
 about 15 seconds: in that time, the SSH sessions to the machine timed out
 and disconnected.  This morning, a machine generated a lot of output to
 the serial console keeping it substantially busy for about 20 seconds; in
 that time, the SSH session to it timed out.  I'm going to see if I can't
 generate some tcpdump traces later today to confirm my suspicions, but was
 wondering if anyone else (annecdotally or not) has seen similar things? 
 
 Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
 [EMAIL PROTECTED]  Network Associates Laboratories

  I have (annecdotally) but I believe I'm seeing it on -STABLE too...
  it's tough to tell... how recent are your -CURRENT machines, though,
  and is it something that you think just started happening or has it
  been happening for a while now?  FWIW, I can't say for sure that this
  is related to TCP connection timeouts.

-- 
Bosko Milekic * [EMAIL PROTECTED] * [EMAIL PROTECTED]


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: TCP connections timing out real fast

[Robert Watson]

On Sat, 22 Feb 2003, Bosko Milekic wrote:

 On Sat, Feb 22, 2003 at 10:57:05AM -0500, Robert Watson wrote:
  
  Don't yet have any quantitative evidence that this is the case, but I feel
  like TCP sessions have been timing out on me a lot faster than they used
  to.  For example, yesterday a machine got unplugged from the network for
  about 15 seconds: in that time, the SSH sessions to the machine timed out
  and disconnected.  This morning, a machine generated a lot of output to
  the serial console keeping it substantially busy for about 20 seconds; in
  that time, the SSH session to it timed out.  I'm going to see if I can't
  generate some tcpdump traces later today to confirm my suspicions, but was
  wondering if anyone else (annecdotally or not) has seen similar things? 
 
   I have (annecdotally) but I believe I'm seeing it on -STABLE too...
   it's tough to tell... how recent are your -CURRENT machines, though,
   and is it something that you think just started happening or has it
   been happening for a while now?  FWIW, I can't say for sure that this
   is related to TCP connection timeouts.

The workstation the sessions originated from is 5.0-RELEASE from Jan 16; 
the build box running sshd is 5.x from Jan 30.  I.e., all before recent
TCP changes.

Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
[EMAIL PROTECTED]  Network Associates Laboratories



To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message

Re: TCP connections timing out real fast

[Robert Watson]

On Sat, 22 Feb 2003, Bosko Milekic wrote:

 On Sat, Feb 22, 2003 at 10:57:05AM -0500, Robert Watson wrote:
  
  Don't yet have any quantitative evidence that this is the case, but I feel
  like TCP sessions have been timing out on me a lot faster than they used
  to.  For example, yesterday a machine got unplugged from the network for
  about 15 seconds: in that time, the SSH sessions to the machine timed out
  and disconnected.  This morning, a machine generated a lot of output to
  the serial console keeping it substantially busy for about 20 seconds; in
  that time, the SSH session to it timed out.  I'm going to see if I can't
  generate some tcpdump traces later today to confirm my suspicions, but was
  wondering if anyone else (annecdotally or not) has seen similar things? 
 
   I have (annecdotally) but I believe I'm seeing it on -STABLE too...
   it's tough to tell... how recent are your -CURRENT machines, though,
   and is it something that you think just started happening or has it
   been happening for a while now?  FWIW, I can't say for sure that this
   is related to TCP connection timeouts.

Here's a packet trace.  cboss.gw.tislabs.com is running the January 30
5.0-CURRENT.  crash2.gw.tislabs.com is running a -CURRENT from yesterday.
Here's the output from the ssh session:

crash2:~ sysctl -a | grep witnessRead from remote host
crash2.gw.tislabs.com: Operation timed out
Connection to crash2.gw.tislabs.com closed.
cboss:/data/stock/src/sys/kern 

The sysctl -a takes a little while to run because it currently generates a
boatload of serial console output due to sleep warnings.  Running it on
the console takes about 35 seconds to complete.  The disconnect
appears to happen half way through that time.  Here's the trace, as
recorded on cboss.gw.tislabs.com, starting about when I hit enter at the
end of the sysctl command line; it looks like it takes about 20 seconds to
decide to disconnect after a series of rapid retransmissions:

cboss# tcpdump -r /tmp/packets
11:40:36.826529 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
347024
1365:3470241385(20) ack 49959986 win 33304 nop,nop,timestamp 25630115
516468 (
DF) [tos 0x10] 
11:40:36.845660 crash2.gw.tislabs.com.ssh  cboss.gw.tislabs.com.49423: P
1:21(2
0) ack 20 win 33304 nop,nop,timestamp 519843 25630115 (DF) [tos 0x10] 
11:40:36.940001 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: .
ack 21
 win 33304 nop,nop,timestamp 25630127 519843 (DF) [tos 0x10] 
11:40:37.758432 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
20:40(
20) ack 21 win 33304 nop,nop,timestamp 25630208 519843 (DF) [tos 0x10] 
11:40:37.775625 crash2.gw.tislabs.com.ssh  cboss.gw.tislabs.com.49423: P
21:41(
20) ack 40 win 33304 nop,nop,timestamp 519936 25630208 (DF) [tos 0x10] 
11:40:37.868677 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: .
ack 41
 win 33304 nop,nop,timestamp 25630220 519936 (DF) [tos 0x10] 
11:40:40.780735 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630511 519936 (DF) [tos 0x10] 
11:40:41.008779 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630534 519936 (DF) [tos 0x10] 
11:40:41.268786 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630560 519936 (DF) [tos 0x10] 
11:40:41.588797 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630592 519936 (DF) [tos 0x10] 
11:40:42.028822 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630636 519936 (DF) [tos 0x10] 
11:40:42.708951 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630704 519936 (DF) [tos 0x10] 
11:40:43.868880 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25630820 519936 (DF) [tos 0x10] 
11:40:45.988960 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25631032 519936 (DF) [tos 0x10] 
11:40:48.109027 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25631244 519936 (DF) [tos 0x10] 
11:40:50.229094 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25631456 519936 (DF) [tos 0x10] 
11:40:52.349177 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25631668 519936 (DF) [tos 0x10] 
11:40:54.469236 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25631880 519936 (DF) [tos 0x10] 
11:40:56.589311 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: P
40:60(
20) ack 41 win 33304 nop,nop,timestamp 25632092 519936 (DF) [tos 0x10] 
11:40:58.709370 cboss.gw.tislabs.com.49423  crash2.gw.tislabs.com.ssh: R
60:60(
0) ack 41 win 33304 

Re: TCP connections timing out real fast

[Giorgos Keramidas]

On 2003-02-22 11:47, Robert Watson [EMAIL PROTECTED] wrote:
 Maybe I'm just too impatient, but it strikes me that I used to get more
 time before TCP gave up during a brief outage.

FWIW, I'm seeing delays in interactive sessions, and lots of timeouts
for `fetchmail -v' runs.  I just started rebuilding a kernel with:

% cd /usr/src/sys
% cvs up -APd -D '2003-02-17 08:00 UTC'

to see if I can track down the change that caused this.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message