I have a problem, and I am unable to find previous discussions of it. Any pointers or clues would be much appreciated.
I have a FreeBSD 5.1 server that needs to be able to handle several thousand simultaneous ssh sessions from distinct users. (I am using FreeBSD 5.1 because I need to be able to support ldap authentication.)
Hardware info: CPU: AMD Athlon(tm) MP 2000+ (1666.74-MHz 686-class CPU) real memory = 1610088448 (1535 MB) avail memory = 1558822912 (1486 MB)
My version of ssh is 3.6.1p2 patched to address the security concerns. (I am not using 3.7.1p because it dropped support for password authentication with PAM, and I cannot assume keyboard-interactive authentication will be present in my users' clients.)
All of these users are doing ssh port forwarding, and are not assigned ptys.
I have not modified login.conf in any way -- the defaults of "no limits" remain.
The kernel tunables in /boot/loader.conf are set to:
kern.maxfiles="49312" kern.maxproc="24656" kern.maxprocperuid="11094" kern.ipc.maxsockets="24656" kern.ipc.somaxconn="8192"
The kernel is compiled with NMBCLUSTERS=65536 and maxusers=0 (which defaults to 384).
The problem is that after about 150 users log in (300ish sshd sessions, since I am using privsep), incoming connections start getting dropped. i.e.,
bash-2.05b# ssh -v localhost
OpenSSH_3.6.1p2, SSH protocols 1.5/2.0, OpenSSL 0x0090701f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: Connecting to localhost [::1] port 22.
socket: Protocol not supported
debug1: Connecting to localhost [127.0.0.1] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host
debug1: Calling cleanup 0x805f010(0x0)
bash-2.05b#
It is my intuition from this behavior that the sshd master process listening for connections is unable to spawn a new process to complete the authentication step, and thus the connection is being dropped. There is no information of use in dmesg, nor in the system logs. (I've cranked up LogLevel to DEBUG3 in sshd_config).
I have a RedHat Linux server running the 2.4.18-3smp kernel on a dual Athlon MP 1800+ and 2048MB RAM that is known to handle 1000 users without issue -- so I have to believe the FreeBSD box, though not as beefy hardware-wise, should be able to do better than a few hundred users. I believe this to be some sort of resource limit issue, but I have addressed everything I could think of.
Here's the sysctl vm.zone output:
vm.zone: ITEM SIZE LIMIT USED FREE REQUESTS
FFS2 dinode: 256, 0, 1089, 21, 1359 FFS1 dinode: 128, 0, 0, 0, 0 FFS inode: 144, 0, 1089, 59, 1359 SWAPMETA: 276, 121576, 0, 0, 0 unpcb: 140, 65548, 329, 63, 31364 ripcb: 228, 49317, 0, 0, 0 syncache: 136, 15370, 0, 58, 36747 tcptw: 48, 49385, 3812, 255, 89831 tcpcb: 360, 49313, 1048, 63, 195072 inpcb: 228, 49317, 4921, 94, 195072 udpcb: 228, 49317, 1, 33, 114497 socket: 256, 49320, 1383, 102, 340934 KNOTE: 64, 0, 0, 124, 114453 PIPE: 176, 0, 622, 68, 17402 DIRHASH: 1024, 0, 138, 6, 138 NAMEI: 1024, 0, 9, 11, 451791 VNODEPOLL: 76, 0, 0, 0, 0 VNODE: 292, 0, 1473, 35, 1473 g_bio: 144, 0, 259, 49, 186276 VMSPACE: 256, 0, 424, 26, 11035 UPCALL: 44, 0, 0, 0, 0 KSE: 64, 0, 496, 62, 496 KSEGRP: 120, 0, 496, 62, 496 THREAD: 292, 0, 496, 11, 496 PROC: 480, 0, 461, 35, 11074 Files: 60, 0, 6051, 153, 89241268 65536: 65536, 0, 3, 3, 3 32768: 32768, 0, 3, 3, 32 16384: 16384, 0, 56, 22, 1733 8192: 8192, 0, 2, 4, 50 4096: 4096, 0, 736, 44, 11965 2048: 2048, 0, 71, 5, 359215 1024: 1024, 0, 408, 20, 284756 512: 512, 0, 102, 18, 43908 256: 256, 0, 5166, 84, 131327 128: 128, 0, 6784, 253, 535182 64: 64, 0, 3032, 68, 87489 32: 32, 0, 2155, 182, 211243 16: 16, 0, 4485, 295, 844397 DP fakepg: 72, 0, 0, 0, 0 PV ENTRY: 28, 5324340, 347293, 45827, 7291251 MAP ENTRY: 60, 0, 27235, 89, 560399 KMAP ENTRY: 60, 32802, 817, 107, 22218 MAP: 176, 0, 8, 38, 6 VM OBJECT: 148, 0, 18751, 122, 318636 UMA Buckets: 512, 0, 741, 1, 0 UMA Hash: 128, 0, 1, 30, 0 UMA Slabs: 34, 0, 1092, 8, 0 UMA Zones: 284, 0, 48, 8, 0
Am I missing anything?
Thanks,
Len
_______________________________________________ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
