Re: Socket leak (Was: Re: What triggers No Buffer Space) ?Available?

2007-05-15 Thread Ulrich Spoerlein
I'm slowly cathing up on FreeBSD related mails and found this mail ... Marc G. Fournier wrote: kern.ipc.numopensockets: 7400 kern.ipc.maxsockets: 12328 ps looks like: stuff deleted 2368 p2 Is+ Sat01PM 0:00.03 /bin/tcsh root2112 0.0 0.1 5220 2360 p3 Ss+

Re: Socket leak (Was: Re: What triggers No Buffer Space) ?Available?

2007-05-15 Thread Jeremy Chadwick
On Tue, May 15, 2007 at 09:39:35PM +0200, Ulrich Spoerlein wrote: How did the backing out work for you? Taken from another mail from Marc, since there's now multiple threads discussing this: Did we determine whether backing out to before the unpcb socket reference count change made any

Re: Socket leak (Was: Re: What triggers No Buffer Space) ?Available?

2007-05-15 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 It didn't kept climbing ... - --On Tuesday, May 15, 2007 21:39:35 +0200 Ulrich Spoerlein [EMAIL PROTECTED] wrote: I'm slowly cathing up on FreeBSD related mails and found this mail ... Marc G. Fournier wrote: kern.ipc.numopensockets:

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-08 Thread Robert Watson
On Tue, 8 May 2007, Marc G. Fournier wrote: So, over 7000 sockets with pretty much all processes shut down ... Shouldn't the garbage collector be cutting in somewhere here? I'm willing to shut everthing down like this again the next time it happens (in 2-3 days) if someone has some other

Re: Socket leak (Was: Re: What triggers No Buffer Space) ?Available?

2007-05-08 Thread Oliver Fromme
Marc G. Fournier wrote: Oliver Fromme wrote: If I remember correctly, you wrote that 11k sockets are in use with 90 jails. That's about 120 sockets per jail, which isn't out of the ordinary. Of course it depends on what is running in those jails, but my guess is that you just

Re: Socket leak (Was: Re: What triggers No Buffer Space) ?Available?

2007-05-08 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Tuesday, May 08, 2007 15:14:29 +0200 Oliver Fromme [EMAIL PROTECTED] wrote: What kind of jails are those? What applications are running inside them? It's quite possible that the processes on one machine use 120 sockets per jail, while

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-07 Thread Oliver Fromme
Marc G. Fournier wrote: Now, that makes sense to me, I can understand that ... but, how would that look as far as netstat -nA shows? Or, would it? For example, I have: You should use -na to list all sockets, not -nA. mars# netstat -nA | grep c9655a20 c9655a20 stream 0 0

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-07 Thread Kris Kennaway
On Mon, May 07, 2007 at 07:01:02PM +0200, Oliver Fromme wrote: Marc G. Fournier wrote: Now, that makes sense to me, I can understand that ... but, how would that look as far as netstat -nA shows? Or, would it? For example, I have: You should use -na to list all sockets, not -nA.

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-07 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Monday, May 07, 2007 19:01:02 +0200 Oliver Fromme [EMAIL PROTECTED] wrote: If I remember correctly, you wrote that 11k sockets are in use with 90 jails. That's about 120 sockets per jail, which isn't out of the ordinary. Of course it

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-04 Thread Ian Smith
On Thu, 3 May 2007, Marc G. Fournier wrote: Robert had mentioned in one of his emails about a Sockets can also exist without any referencing process (if the application closes, but there is still data draining on an open socket). [..] Again, if I'm reading / understanding things

Re: What triggers No Buffer Space Available?

2007-05-04 Thread Robert Watson
On Thu, 3 May 2007, Marc G. Fournier wrote: 'k, all I'm looking at right now is the Unix Domain Sockets, and the output of netstat - sockstat is growing since I first started counting both .. This was shortly after reboot: mars# netstat -A | grep stream | wc -l ; sockstat -u | wc -l 2705

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-04 Thread Robert Watson
On Thu, 3 May 2007, Marc G. Fournier wrote: I'm trying to probe this as well as I can, but network stacks and sockets have never been my strong suit ... Robert had mentioned in one of his emails about a Sockets can also exist without any referencing process (if the application closes, but

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-04 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Friday, May 04, 2007 12:05:11 +0100 Robert Watson [EMAIL PROTECTED] wrote: I think we should be careful to avoid prematurely drawing conclusions about the source of the problem. First question: have you confirmed that the resource limit

Re: What triggers No Buffer Space Available?

2007-05-03 Thread Chuck Swiger
Marc G. Fournier wrote: [ ... ] okay, next question ... under 'Active UNIX domain sockets, I see alot that have no Addr: Active UNIX domain sockets Address Type Recv-Q Send-QInode Conn Refs Nextref Addr d06b7480 stream 0 00 c969b24000

Re: What triggers No Buffer Space Available?

2007-05-03 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Thursday, May 03, 2007 11:17:56 -0400 Chuck Swiger [EMAIL PROTECTED] wrote: The ones you're showing are from Postfix. It would be interesting to sort them by frequency and see what the majority of the use is from. If you sort the data

Re: What triggers No Buffer Space Available?

2007-05-03 Thread Robert Watson
On Tue, 1 May 2007, Marc G. Fournier wrote: I'm still being hit by this one ... more frequently right now as I had to move a bit more stuff *onto* that server ... I'm trying to figure out what I can monitor for a 'leak' somewhere, but the only thing I'm able to find is the whole nmbclusters

Re: What triggers No Buffer Space Available?

2007-05-03 Thread Robert Watson
On Wed, 2 May 2007, Marc G. Fournier wrote: # netstat | egrep tcp4|udp4 | awk '{print $1}' | uniq -c 171 tcp4 103 udp4 or is there a better command I should be using? I generally recommend using a combination of netstat and sockstat. Sockets represent, loosely, IPC endpoints. There are

Re: What triggers No Buffer Space Available?

2007-05-03 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Thursday, May 03, 2007 19:28:56 +0100 Robert Watson [EMAIL PROTECTED] wrote: I generally recommend using a combination of netstat and sockstat. Sockets represent, loosely, IPC endpoints. There are actually two layers associated with

Re: What triggers No Buffer Space Available?

2007-05-03 Thread Adrian Chadd
On 04/05/07, Marc G. Fournier [EMAIL PROTECTED] wrote: 'k, all I'm looking at right now is the Unix Domain Sockets, and the output of netstat - sockstat is growing since I first started counting both .. Hm! What about graphing them? It shouldn't be hard to write an mrtg shell script data

Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-03 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm trying to probe this as well as I can, but network stacks and sockets have never been my strong suit ... Robert had mentioned in one of his emails about a Sockets can also exist without any referencing process (if the application closes, but

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-03 Thread Matthew Dillon
:I'm trying to probe this as well as I can, but network stacks and sockets have :never been my strong suit ... : :Robert had mentioned in one of his emails about a Sockets can also exist :without any referencing process (if the application closes, but there is still :data draining on an open

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-03 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Thursday, May 03, 2007 18:26:30 -0700 Matthew Dillon [EMAIL PROTECTED] wrote: One thing you can do is drop into single user mode... kill all the processes on the system, and see if the sockets are recovered. That will give

Re: Socket leak (Was: Re: What triggers No Buffer Space) Available?

2007-05-03 Thread Matthew Dillon
:*groan* why couldn't this be happening on a server that I have better remote :access to? :( : :But, based on your explanation(s) above ... if I kill off all of the jail(s) on :the machine, so that there are minimal processes running, shouldn't I see a :significant drop in the number of

Re: What triggers No Buffer Space Available?

2007-05-02 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Wednesday, May 02, 2007 11:00:17 +0800 Adrian Chadd [EMAIL PROTECTED] wrote: It doesn't panic whe it happens, no? Nope ... I can login via ssh (sometimes it takes a try or two, but I can always login) and then do a 'reboot', and all is

Re: What triggers No Buffer Space Available?

2007-05-02 Thread John-Mark Gurney
Marc G. Fournier wrote this message on Wed, May 02, 2007 at 14:34 -0300: Is there any way of determining which apps are holding open which sockets? ie. lsof for open files? netstat -A will list the socket address, fstat will list the fd, and what socket it connected to that fd.. --

Re: What triggers No Buffer Space Available?

2007-05-02 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - --On Wednesday, May 02, 2007 11:17:02 -0700 John-Mark Gurney [EMAIL PROTECTED] wrote: netstat -A will list the socket address, fstat will list the fd, and what socket it connected to that fd.. Oh wow ... according to this, I have: mars# wc

Re: What triggers No Buffer Space Available?

2007-05-02 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 'k, I just rebooted the server (messages started again), and netstat -A is showing 3600 sockets open ... based on jupiter/pluto/venus numbers, this is what I'd expect to see (~1000 sockets per 30 jails) ... so, over the course of hte next 2 days,

What triggers No Buffer Space Available?

2007-05-01 Thread Marc G. Fournier
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm still being hit by this one ... more frequently right now as I had to move a bit more stuff *onto* that server ... I'm trying to figure out what I can monitor for a 'leak' somewhere, but the only thing I'm able to find is the whole nmbclusters

Re: What triggers No Buffer Space Available?

2007-05-01 Thread Adrian Chadd
On 01/05/07, Marc G. Fournier [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I'm still being hit by this one ... more frequently right now as I had to move a bit more stuff *onto* that server ... I'm trying to figure out what I can monitor for a 'leak' somewhere, but