On Fri, Mar 24, 2006 at 06:03:47PM -0500, Charles Swiger wrote:
> On Mar 24, 2006, at 4:17 PM, Michael W. Lucas wrote:
> >Running FreeBSD 6.1-PRERELEASE as a DNS, dhcp, and syslog server.
> >
> >I'm having trouble with DNS, DHCP, and syslogd locking up, and I think
> >I've found what they all share in common.
> >
> >During the lockups, the box starts dropping UDP due to full socket
> >buffers.  I have a dumb little script to capture the rate of drops
> >over 5 seconds, and it's about 45 a second.
> >
> >168725 dropped due to full socket buffers
> >168958 dropped due to full socket buffers
> 
> There is generally a cause behind the socket buffers filling up,  
> whether that is some form of livelock due to an OS problem or a  
> misconfiguration with a firewall/dummynet setup.  You could look at  
> the output of "netstat -a(n)" for insight as to where the packets are  
> being queued up, but "netstat -s" would be useful to show to us as well.

Thanks.  I think you've shown me how to find the problem:

# netstat -na
...
udp4       0      0  127.0.0.1.57058        127.0.0.1.53           
udp4       0      0  127.0.0.1.61259        127.0.0.1.53           
udp4       0      0  127.0.0.1.54240        127.0.0.1.53           
udp4       0      0  127.0.0.1.52997        127.0.0.1.53           
udp4       0      0  *.67                   *.*                    
udp4   43414      0  *.514                  *.*                    
udp4       0      0  *.49661                *.*    
...

We have no firewall on this machine; it's buried behind three layers
of firewall.

I've tried running syslogd in debug mode, but not found anything
particularly useful yet.  Syslogd is now set to restart every 15
minutes, and run in debug mode, so hopefully the next time this
happens I'll have the debugging output.  The problem happens even
within fifteen minutes, but because of my timeouts nobody notices.

I'm attaching the output of netstat -na and netstat -s for general
informative purposes; if anyone has any further suggestions, I'm all
ears.

Thanks,
==ml

-- 
Michael W. Lucas        [EMAIL PROTECTED], [EMAIL PROTECTED]
                http://www.BlackHelicopters.org/~mwlucas/

"The cloak of anonymity protects me from the nuisance of caring." -Non Sequitur




Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4      96      0  10.184.1.11.53         10.184.1.8.49838       ESTABLISHED
tcp4     104      0  10.184.1.11.53         10.184.1.8.61719       ESTABLISHED
tcp4       0      0  127.0.0.1.6011         *.*                    LISTEN
tcp4       0     48  10.184.1.11.22         10.184.184.108.52392   ESTABLISHED
tcp4       0      0  127.0.0.1.6010         *.*                    LISTEN
tcp4       0      0  10.184.1.11.22         10.184.184.108.64567   ESTABLISHED
tcp4       0      0  *.199                  *.*                    LISTEN
tcp4       0      0  *.80                   *.*                    LISTEN
tcp4       0      0  127.0.0.1.25           *.*                    LISTEN
tcp4       0      0  *.22                   *.*                    LISTEN
tcp4       0      0  127.0.0.1.953          *.*                    LISTEN
tcp4       0      0  127.0.0.1.53           *.*                    LISTEN
tcp4       0      0  10.184.3.248.53        *.*                    LISTEN
tcp4       0      0  10.184.1.11.53         *.*                    LISTEN
udp4       0      0  127.0.0.1.59578        127.0.0.1.53           
udp4       0      0  127.0.0.1.57058        127.0.0.1.53           
udp4       0      0  127.0.0.1.61259        127.0.0.1.53           
udp4       0      0  127.0.0.1.54240        127.0.0.1.53           
udp4       0      0  127.0.0.1.52997        127.0.0.1.53           
udp4       0      0  *.67                   *.*                    
udp4   43414      0  *.514                  *.*                    
udp4       0      0  *.49661                *.*                    
udp4       0      0  *.52767                *.*                    
udp4       0      0  *.63357                *.*                    
udp4       0      0  127.0.0.1.58915        127.0.0.1.5680         
udp4       0      0  127.0.0.1.62275        127.0.0.1.5679         
udp4       0      0  127.0.0.1.57431        127.0.0.1.5678         
udp4       0      0  *.63077                *.*                    
udp4       0      0  *.161                  *.*                    
udp4       0      0  *.57589                *.*                    
udp4       0      0  *.5681                 *.*                    
udp4       0      0  127.0.0.1.5680         *.*                    
udp4       0      0  127.0.0.1.5679         *.*                    
udp4       0      0  127.0.0.1.5678         *.*                    
udp4       0      0  *.64486                *.*                    
udp4       0      0  *.53558                *.*                    
udp4       0      0  127.0.0.1.123          *.*                    
udp4       0      0  10.184.3.248.123       *.*                    
udp4       0      0  10.184.1.11.123        *.*                    
udp4       0      0  *.123                  *.*                    
udp4     909      0  *.58684                *.*                    
udp4    4129      0  127.0.0.1.53           *.*                    
udp4     278      0  10.184.3.248.53        *.*                    
udp4    8436      0  10.184.1.11.53         *.*                    
icm4       0      0  *.*                    *.*                    
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
c332d690 stream      0      0 c31c4bb0        0        0        0 
/tmp/ssh-GceIKkNhTt/agent.1161
c332d71c stream      0      0        0 c332d460        0        0
c332d460 stream      0      0        0 c332d71c        0        0
c332dc08 stream      0      0 c31aa770        0        0        0 
/tmp/ssh-YFEQdkNEm5/agent.1148
c332d8c0 stream      0      0        0 c332de38        0        0
c332de38 stream      0      0        0 c332d8c0        0        0
c332d08c stream      0      0 c3667330        0        0        0 
/var/run/services.softflowd.ctl
c332d578 stream      0      0 c3667660        0        0        0 
/var/run/mpls.softflowd.ctl
c332d604 stream      0      0 c3667990        0        0        0 
/var/run/anx.softflowd.ctl
c332e000 stream      0      0 c3146cc0        0        0        0 
/var/run/devd.pipe
c332d834 dgram       0      0        0 c332dc94        0 c332d7a8
c332d7a8 dgram       0      0        0 c332dc94        0 c332d118
c332d118 dgram       0      0        0 c332dc94        0 c332db7c
c332d2bc dgram       0      0        0 c332d348        0        0
c332db7c dgram       0      0        0 c332dc94        0        0
c332d348 dgram    4048      0 c32a5660        0 c332d2bc        0 
/var/named/var/run/log
c332dd20 dgram       0      0 c32a5770        0        0        0 /var/run/log
c332dc94 dgram    4048      0 c32a5880        0 c332d834        0 
/var/run/logpriv
c332d000 dgram       0      0 c32a5990        0        0        0 /var/run/log
c332dec4 dgram       0      0        0        0        0 c332d7a8
c332d4ec dgram       0      0        0        0        0 c332d8c0
c332d9d8 dgram       0      0        0        0        0        0
c332da64 dgram       0      0        0        0        0 c332db7c
tcp:
        1986 packets sent
                1254 data packets (124262 bytes)
                0 data packets (0 bytes) retransmitted
                0 data packets unnecessarily retransmitted
                0 resends initiated by MTU discovery
                486 ack-only packets (35 delayed)
                0 URG only packets
                0 window probe packets
                0 window update packets
                246 control packets
        2428 packets received
                1587 acks (for 124643 bytes)
                23 duplicate acks
                0 acks for unsent data
                1142 packets (79497 bytes) received in-sequence
                0 completely duplicate packets (0 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                0 out-of-order packets (0 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                8 window update packets
                0 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        25 connection requests
        208 connection accepts
        2 bad connection attempts
        0 listen queue overflows
        0 ignored RSTs in the windows
        225 connections established (including accepts)
        232 connections closed (including 0 drops)
                26 connections updated cached RTT on close
                26 connections updated cached RTT variance on close
                0 connections updated cached ssthresh on close
        8 embryonic connections dropped
        1392 segments updated rtt (of 1373 attempts)
        0 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 keepalive timeouts
                0 keepalive probes sent
                0 connections dropped by keepalive
        187 correct ACK header predictions
        525 correct data packet header predictions
        208 syncache entries added
                0 retransmitted
                0 dupsyn
                0 dropped
                208 completed
                0 bucket overflow
                0 cache overflow
                0 reset
                0 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures
        0 cookies sent
        0 cookies received
        0 SACK recovery episodes
        0 segment rexmits in SACK recovery episodes
        0 byte rexmits in SACK recovery episodes
        0 SACK options (SACK blocks) received
        0 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
udp:
        550908 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        33451 dropped due to no socket
        63890 broadcast/multicast datagrams dropped due to no socket
        63147 dropped due to full socket buffers
        0 not for hashed pcb
        390420 delivered
        276481 datagrams output
ip:
        560670 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        554968 packets for this host
        42 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        3059 packets not forwardable
        2601 packets received for unknown multicast group
        0 redirects sent
        312928 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        33294 calls to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                echo reply: 1088
                destination unreachable: 33294
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echo reply: 16
                destination unreachable: 261
                routing redirect: 309
                echo: 1088
        1088 message responses generated
        0 invalid return addresses
        0 no return routes
        ICMP address mask responses are disabled
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 membership reports sent
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"

Reply via email to