Re: post-install of kernal sources, maxusers max?
On Wed, 8 Nov 2000, Len Conrad wrote: Sorry to bother you hackers, but -questions isn't responding, and the handbook and Complete/Lehey don't, afaics, cover this situation explicitly. I can't really afford to screw up this production machine and start over from fresh disk, nor futz around for hours guessing what magik combo of post-install choices will do the trick. == I'm working, remotely, on a 4.1 system with only a binary install from cdrom. Now I need to do a custom kernal. Can the /stand/systinstall post-config option be used to put on all the developer source pkg without bothering the current config? which choice (I don't want X, just enough to build a custom kernal) Install cvsup-binary from ports, and cvsup the sources. Leif To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
post-install of kernal sources, maxusers max?
Sorry to bother you hackers, but -questions isn't responding, and the handbook and Complete/Lehey don't, afaics, cover this situation explicitly. I can't really afford to screw up this production machine and start over from fresh disk, nor futz around for hours guessing what magik combo of post-install choices will do the trick. == I'm working, remotely, on a 4.1 system with only a binary install from cdrom. Now I need to do a custom kernal. Can the /stand/systinstall post-config option be used to put on all the developer source pkg without bothering the current config? which choice (I don't want X, just enough to build a custom kernal) It's in production as 200 K msgs/day mail hub. All I need to change, I think, is maxusers since we're getting this error from postfix: Nov 8 04:59:41 postfix/qmgr[16383]: fatal: socket: No buffer space available Nov 8 04:59:41 postfix/smtp[16872]: fatal: socket: No buffer space available Nov 8 05:00:58 postfix/qmgr[16876]: fatal: socket: No buffer space available The machine can get up 200 SMTP processes and 50 SMTPD processes simulatenously, 256 meg RAM. Increasing maxusers will fix this pb? afaic, maxusers can't be fixed with sysctl. Also, having been over /stand/systinstall a few times in last hour, there is no post-install choice that corresponds to an initial install "developer's sources less X and games", which is what I do when I install (this customer didn't follow my instructions for the initial insall). Also: On the postfix list, it seems someone has heard from several FreeBSD "experts" that FreeBSD should not be run at above maxusers = 128, while somebody else said they were running with 256 happily. Comments, please? Len http://BIND8NT.MEIway.com: ISC BIND 8.2.2 p5 8.2.3 T6B for NT4 W2K http://IMGate.MEIway.com: Build free, hi-perf, anti-spam mail gateways To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
On Wed, 8 Nov 2000, Len Conrad wrote: All I need to change, I think, is maxusers since we're getting this error from postfix: Nov 8 04:59:41 postfix/qmgr[16383]: fatal: socket: No buffer space available Nov 8 04:59:41 postfix/smtp[16872]: fatal: socket: No buffer space available Nov 8 05:00:58 postfix/qmgr[16876]: fatal: socket: No buffer space available The machine can get up 200 SMTP processes and 50 SMTPD processes simulatenously, 256 meg RAM. Increasing maxusers will fix this pb? afaic, maxusers can't be fixed with sysctl. I think you can up the mbuf related settings while the system is running. Give it a try. The two sysctls you'll want to fiddle with are: kern.ipc.nmbclusters kern.ipc.nmbufs You can determine which is needed more through a quick netstat -m. Mike "Silby" Silbersack To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
In message [EMAIL PROTECTED], Len Conrad writes: All I need to change, I think, is maxusers since we're getting this error from postfix: You may be able to increase these limits without recompiling the kernel, by using kernel environment variables set in /boot/loader.rc. First of all you need to find out exactly what resource is getting exhausted. I suspect it is the socket space memory zone, but you can verify this with the commands: vmstat -z |grep ^socket # vmstat -z ZONEusedtotal mem-use PIPE36 1025/15K SWAPMETA8 50 1/7K tcpcb 179 350 95/185K unpcb 418 896 26/56K ripcb 0 42 0/7K tcpcb 0 0 0/0K udpcb 10 1681/31K socket 607 1050 113/196K KNOTE 6 2560/16K NFSNODE 0 0 0/0K NFSMOUNT0 0 0/0K VNODE 40064028 751/755K NAMEI 0 40 0/40K VMSPACE 260 384 48/72K PROC264 343 107/139K DP fakepg 0 0 0/0K PV ENTRY53272 79979 1456/2186K MAP ENTRY 31173953 146/185K KMAP ENTRY 804 978 37/45K MAP 7 10 0/1K VM OBJECT 49075464 460/512K -- TOTAL 3252/4455K Do you see anything else up there that looks bad or über limit? sysctl kern.ipc.maxsockets # sysctl kern.ipc.maxsockets kern.ipc.maxsockets: 1064 If the 'max' column in the vmstat output matches the number returned by sysctl, then the kernel has hit this limit at some stage. doesn't look like it to me. To increase the limit, pick a new larger value, and add a line to /boot/loader.rc e.g. set kern.ipc.maxsockets=5000 You will need to reboot for this setting to take effect, and it is not possible to dynamically increase the limit in a running kernel. Other settings worth investigating are nmbclusters and nmbufs. These are viewable via 'netstat -m', and can be set as above. # netstat -m 358/960/4096 mbufs in use (current/peak/max): 198 mbufs allocated to data 160 mbufs allocated to packet headers 185/504/1024 mbuf clusters in use (current/peak/max) 1248 Kbytes allocated to network (36% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines So what do you think postfix is complaining about, more precisely, how to fix it: Fatal Errors (postfix) bounce 1 socket: No buffer space available qmgr 7 socket: No buffer space available script 1 usage: postfix start (or stop, reload, abort, flush, or check) smtp 5 socket: No buffer space available 3 inet_addr_local: socket: No buffer space available smtpd 6 socket: No buffer space available Thanks, Len To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
On Wed, 8 Nov 2000, Mike Silbersack wrote: The machine can get up 200 SMTP processes and 50 SMTPD processes simulatenously, 256 meg RAM. Increasing maxusers will fix this pb? afaic, maxusers can't be fixed with sysctl. I think you can up the mbuf related settings while the system is running. Give it a try. The two sysctls you'll want to fiddle with are: kern.ipc.nmbclusters kern.ipc.nmbufs Nope. Those are read only at least on my 4.2-BETA kernel. I am running into the same problem during my benchmarking. I also thought that a change in the past made the mbuf cluters tuneable on a running machine. Hmm maybe I am wrong. However Im not sure that is the problem. I have: [open-systems]:/home/chris netstat -m 5/2000/6144 mbufs in use (current/peak/max): 1 mbufs allocated to data 4 mbufs allocated to packet headers 0/1096/1536 mbuf clusters in use (current/peak/max) I still have free mbuf's yet get the same error trying to make postfix bleed to death: [open-systems]:/home/chris /usr/bin/time -p smtp-source -s 1 -l 10 -m 1000 -c -f chris@localhost -t null@localhost localhost smtp-source: fatal: socket: No buffer space available I also get: [open-systems]:/home/chris /usr/bin/time -p smtp-source -s 1 -l 10 -m 1000 -c -f chris@localhost -t null@localhost localhost smtp-source: fatal: lost connection while sending sender I am working on figuring out what the magic combo is to get postfix to really doll out connections on FreeBSD. = -Chris Watson (316) 326-3862 | FreeBSD Consultant, FreeBSD Geek Work: [EMAIL PROTECTED] | Open Systems Inc., Wellington, Kansas Home: [EMAIL PROTECTED] | http://open-systems.net = WINDOWS: "Where do you want to go today?" LINUX: "Where do you want to go tommorow?" BSD: "Are you guys coming or what?" = irc.openprojects.net #FreeBSD -Join the revolution! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
In message [EMAIL PROTECTED], Len Conrad writes: # vmstat -z ... socket 607 1050 113/196K ... kern.ipc.maxsockets: 1064 doesn't look like it to me. I think a few slots are reserved, so you can consider 1050 as being equal to 1064. Try putting set kern.ipc.maxsockets=4000 in /boot/loader.rc and rebooting. Ian To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
kern.ipc.nmbclusters kern.ipc.nmbufs # sysctl -w kern.ipc.nmbclusters=2048 sysctl: oid 'kern.ipc.nmbclusters' is read only # sysctl -w kern.ipc.nmbufs=8192 sysctl: oid 'kern.ipc.nmbufs' is read only I'll have to reboot, You can determine which is needed more through a quick netstat -m. # netstat -m 367/960/4096 mbufs in use (current/peak/max): 189 mbufs allocated to data 178 mbufs allocated to packet headers 150/504/1024 mbuf clusters in use (current/peak/max) 1248 Kbytes allocated to network (31% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines question still is what postfix is missing. I guess I better try to postfix list again. Thanks, Silby Len To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
kern.ipc.nmbclusters kern.ipc.nmbufs Nope. Those are read only at least on my 4.2-BETA kernel. read-only also in 4.1 # sysctl -w kern.ipc.nmbclusters=2048 sysctl: oid 'kern.ipc.nmbclusters' is read only # sysctl -w kern.ipc.nmbufs=8192 sysctl: oid 'kern.ipc.nmbufs' is read only I am running into the same problem during my benchmarking. I also thought that a change in the past made the mbuf cluters tuneable on a running machine. Hmm maybe I am wrong. However Im not sure that is the problem. I have: [open-systems]:/home/chris netstat -m 5/2000/6144 mbufs in use (current/peak/max): 1 mbufs allocated to data 4 mbufs allocated to packet headers 0/1096/1536 mbuf clusters in use (current/peak/max) I still have free mbuf's yet get the same error trying to make postfix bleed to death: [open-systems]:/home/chris /usr/bin/time -p smtp-source -s 1 -l 10 -m 1000 -c -f chris@localhost -t null@localhost localhost smtp-source: fatal: socket: No buffer space available yep, me, too, can't see exactly what postfix is bitching about: # netstat -m 367/960/4096 mbufs in use (current/peak/max): 189 mbufs allocated to data 178 mbufs allocated to packet headers 150/504/1024 mbuf clusters in use (current/peak/max) 1248 Kbytes allocated to network (31% in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines If you learn something, let me know, please. Thanks Len To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
* Len Conrad [EMAIL PROTECTED] [001108 12:29] wrote: Sorry to bother you hackers, but -questions isn't responding, and the handbook and Complete/Lehey don't, afaics, cover this situation explicitly. I can't really afford to screw up this production machine and start over from fresh disk, nor futz around for hours guessing what magik combo of post-install choices will do the trick. == I'm working, remotely, on a 4.1 system with only a binary install from cdrom. Now I need to do a custom kernal. Can the /stand/systinstall kernel! :) post-config option be used to put on all the developer source pkg without bothering the current config? which choice (I don't want X, just enough to build a custom kernal) It's in production as 200 K msgs/day mail hub. All I need to change, I think, is maxusers since we're getting this error from postfix: Nov 8 04:59:41 postfix/qmgr[16383]: fatal: socket: No buffer space available Nov 8 04:59:41 postfix/smtp[16872]: fatal: socket: No buffer space available Nov 8 05:00:58 postfix/qmgr[16876]: fatal: socket: No buffer space available The machine can get up 200 SMTP processes and 50 SMTPD processes simulatenously, 256 meg RAM. Increasing maxusers will fix this pb? afaic, maxusers can't be fixed with sysctl. Yes, but nmbclusters can, see the loader(8) manpage for the tunables to raise kern.ipc.nmbclusters, you might have better luck setting it to 32768. Also: On the postfix list, it seems someone has heard from several FreeBSD "experts" that FreeBSD should not be run at above maxusers = 128, while somebody else said they were running with 256 happily. There was a very short period where FreeBSD 3.x wouldn't work properly when maxusers was above 256, but that hasn't been an issue for quite some time. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
On Wed, 8 Nov 2000, Alfred Perlstein wrote: Yes, but nmbclusters can, see the loader(8) manpage for the tunables to raise kern.ipc.nmbclusters, you might have better luck setting it to 32768. Is it possible to make the tuning of nmbclusters available after the kenrel is loaded. So that you don't have to reboot a server to get loader's changes to take effect? when maxusers was above 256, but that hasn't been an issue for quite some time. So one could go as high as.. 512? 1024? There has to still be drawbacks at some number where your wasting resources that you dont need just to get more mbuf's. I think that is why they are saying dont just jack up MAXUSERS. Use the NMBCLUSTERS= instead. Because that is usually the variable you want increased not the other parameters MAXUSERS increases. = -Chris Watson (316) 326-3862 | FreeBSD Consultant, FreeBSD Geek Work: [EMAIL PROTECTED] | Open Systems Inc., Wellington, Kansas Home: [EMAIL PROTECTED] | http://open-systems.net = WINDOWS: "Where do you want to go today?" LINUX: "Where do you want to go tommorow?" BSD: "Are you guys coming or what?" = irc.openprojects.net #FreeBSD -Join the revolution! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
* [EMAIL PROTECTED] [EMAIL PROTECTED] [001108 14:22] wrote: On Wed, 8 Nov 2000, Alfred Perlstein wrote: Yes, but nmbclusters can, see the loader(8) manpage for the tunables to raise kern.ipc.nmbclusters, you might have better luck setting it to 32768. Is it possible to make the tuning of nmbclusters available after the kenrel is loaded. So that you don't have to reboot a server to get loader's changes to take effect? Nope. when maxusers was above 256, but that hasn't been an issue for quite some time. So one could go as high as.. 512? 1024? There has to still be drawbacks at some number where your wasting resources that you dont need just to get more mbuf's. I think that is why they are saying dont just jack up MAXUSERS. Use the NMBCLUSTERS= instead. Because that is usually the variable you want increased not the other parameters MAXUSERS increases. I've never had to set maxusers higher than 512 and 256 would have probably been fine. -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
"scanner" == [EMAIL PROTECTED] writes: scannerSo one could go as high as.. 512? 1024? There has to scanner still be drawbacks at some number where your wasting scanner resources that you dont need just to get more mbuf's. I scanner think that is why they are saying dont just jack up scanner MAXUSERS. Use the NMBCLUSTERS= instead. Because that scanner is usually the variable you want increased not the other scanner parameters MAXUSERS increases. FWIW I run our NFS server with NMBCLUSTERS=1. It doesn't burn that much additional memory. --lyndon To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
On Wed, 8 Nov 2000, Mike Silbersack wrote: I think you can up the mbuf related settings while the system is running. Give it a try. The two sysctls you'll want to fiddle with are: kern.ipc.nmbclusters kern.ipc.nmbufs Nope. These are read-only but can be tuned from loader. You can determine which is needed more through a quick netstat -m. Mike "Silby" Silbersack Cheers, Bosko Milekic [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
Lyndon Nerenberg writes: FWIW I run our NFS server with NMBCLUSTERS=1. It doesn't burn that much additional memory. As an additional data point, I had an NFS server that regularly crashed when it ran out; logs showed that it needed up to 1700 (against the default of 1024). I bumped it to 4096 about a year ago to give myself a bit of headroom and have no problems since; and I have noticed no waste of resources that mattered. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
On Wed, 8 Nov 2000, Len Conrad wrote: All I need to change, I think, is maxusers since we're getting this error from postfix: Nov 8 04:59:41 postfix/qmgr[16383]: fatal: socket: No buffer space available Nov 8 04:59:41 postfix/smtp[16872]: fatal: socket: No buffer space available Nov 8 05:00:58 postfix/qmgr[16876]: fatal: socket: No buffer space available You're running out of send/recvspace or your ethernet is maxed out. net.inet.tcp.sendspace: 16384 net.inet.tcp.recvspace: 16384 Try tweaking these values. You should also bump maxusers/nmbcluters/nmbufs so you're using about 20-30% nominally. Also make sure you're using full duplex links and you're not seeing excessive errors/collisions. I'd have to watch 'systat -vm' to check for some of the more remote problems ... The machine can get up 200 SMTP processes and 50 SMTPD processes simulatenously, 256 meg RAM. Small processes. On the postfix list, it seems someone has heard from several FreeBSD "experts" that FreeBSD should not be run at above maxusers = 128, while somebody else said they were running with 256 happily. Depends on the release... I would not go above 256. Doug White| FreeBSD: The Power to Serve [EMAIL PROTECTED] | www.FreeBSD.org To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: post-install of kernal sources, maxusers max?
Ian Dowse wrote: I think a few slots are reserved, so you can consider 1050 as being equal to 1064. Try putting set kern.ipc.maxsockets=4000 in /boot/loader.rc and rebooting. Eeee! kern.ipc.maxsockets="4000" in /boot/loader.conf instead, please! -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] He has been convicted of criminal possession of a clue with intent to distribute. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
