This diff fixed it, for anyone curious, I don't know why a cvsup didnt get
it:
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/uipc_socket2.c.diff?r1=1.
78&r2=1.79&f=h
--IS
----- Original Message -----
From: "ISPrime Support" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, December 25, 2001 2:53 AM
Subject: 4.5-PRERELEASE Major Stalling
> I work at an ISP which hosts zfreehost.com (a pretty large freehost,
pushing
> about 300 megabit).
> The zfreehost network has a few FBSD servers (all 4.4-STABLE except one
> 4.5-PRERELEASE) running a custom in-house webserver, using a FBSD NFS
> Storage backend.
> Anyway, I upgraded one of the servers to 4.5-PRERELEASE just to test it
out
> for us, and I noticed there are MAJOR stalls in 4.5. After about 5
minutes
> of running, all of the webserver processes (8 processes) are sitting in
> state sbwait, doing nothing. in a truss, it like it stalls on the read()
> immediately after accept() from the client socket (http). We run httpready
> accept filtering, so when we do a read() it should never ever stall since
it
> wouldn't have accept()'d if there wasnt data to read(), or if it was ready
> to close(), anyway -- I consider this to be a pretty major regression.
At
> the way this is running, we wont be able to run 4.5 on any of the
production
> servers, so I hope this can get fixed.
> Here is what I see from top:
> PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
> 266 nobody 2 0 992K 664K sbwait 0:01 0.63% 0.63% pimpd
> 267 nobody 2 0 992K 664K sbwait 0:01 0.10% 0.10% pimpd
> 264 nobody 2 0 992K 664K sbwait 0:01 0.05% 0.05% pimpd
> 86 root 10 0 208K 80K nfsidl 0:00 0.05% 0.05% nfsiod
> 123 root 28 0 2348K 1944K RUN 0:01 0.00% 0.00% sshd
> 85 root 10 0 208K 80K nfsidl 0:01 0.00% 0.00% nfsiod
> 269 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
> 262 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
> 268 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
> 263 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
> 265 nobody 2 0 992K 664K sbwait 0:00 0.00% 0.00% pimpd
>
> This is what I see from ps auwx:
> nobody 262 0.0 0.1 992 664 ?? Is 2:47AM 0:00.37 pimpd
> nobody 263 0.0 0.1 992 664 ?? I 2:47AM 0:00.25 pimpd
> nobody 264 0.0 0.1 992 664 ?? I 2:47AM 0:01.19 pimpd
> nobody 265 0.0 0.1 992 664 ?? I 2:47AM 0:00.13 pimpd
> nobody 266 0.0 0.1 992 664 ?? I 2:47AM 0:01.74 pimpd
> nobody 267 0.0 0.1 992 664 ?? I 2:47AM 0:01.95 pimpd
> nobody 268 0.0 0.1 992 664 ?? I 2:47AM 0:00.30 pimpd
> nobody 269 0.0 0.1 992 664 ?? I 2:47AM 0:01.96 pimpd
>
> I did not enable fxp bundling.
>
> Here is sysctl -a:
> kern.ostype: FreeBSD
> kern.osrelease: 4.5-PRERELEASE
> kern.osrevision: 199506
> kern.version: FreeBSD 4.5-PRERELEASE #0: Tue Dec 25 02:13:41 EST 2001
> root@zf3:/usr/src/sys/compile/ZFREE
>
> kern.maxvnodes: 68412
> kern.maxproc: 4116
> kern.maxfiles: 8232
> kern.argmax: 65536
> kern.securelevel: -1
> kern.hostname: zfreeVA3.isprime.com
> kern.hostid: 0
> kern.clockrate: { hz = 100, tick = 10000, tickadj = 5, profhz = 1024,
stathz
> = 128 }
> kern.posix1version: 199309
> kern.ngroups: 16
> kern.job_control: 1
> kern.saved_ids: 0
> kern.boottime: { sec = 1009265636, usec = 169977 } Tue Dec 25 02:33:56
2001
> kern.domainname:
> kern.osreldate: 450000
> kern.bootfile: /kernel
> kern.maxfilesperproc: 7408
> kern.maxprocperuid: 3704
> kern.dumpdev:
> kern.ipc.maxsockbuf: 2097152
> kern.ipc.sockbuf_waste_factor: 8
> kern.ipc.somaxconn: 10240
> kern.ipc.max_linkhdr: 16
> kern.ipc.max_protohdr: 40
> kern.ipc.max_hdr: 56
> kern.ipc.max_datalen: 156
> kern.ipc.nmbclusters: 65536
> kern.ipc.semmap: 30
> kern.ipc.semmni: 10
> kern.ipc.semmns: 60
> kern.ipc.semmnu: 30
> kern.ipc.semmsl: 60
> kern.ipc.semopm: 100
> kern.ipc.semume: 10
> kern.ipc.semusz: 92
> kern.ipc.semvmx: 32767
> kern.ipc.semaem: 16384
> kern.ipc.shmmax: 409600000
> kern.ipc.shmmin: 1
> kern.ipc.shmmni: 192
> kern.ipc.shmseg: 128
> kern.ipc.shmall: 8192
> kern.ipc.shm_use_phys: 0
> kern.ipc.mbuf_wait: 32
> kern.ipc.mbtypes: 3589 3759 76 0 0 0 0 0 0 0 0 0 0 0 0 0
> kern.ipc.nmbufs: 262144
> kern.ipc.maxsockets: 65536
> kern.dummy: 0
> kern.ps_strings: 3217031152
> kern.usrstack: 3217031168
> kern.logsigexit: 1
> kern.cam.da.retry_count: 4
> kern.cam.da.default_timeout: 60
> kern.fallback_elf_brand: -1
> kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall
> kern.module_path: /;/boot/;/modules/
> kern.acct_suspend: 2
> kern.acct_resume: 4
> kern.acct_chkfreq: 15
> kern.cp_time: 511 0 2075 2732 147211
> kern.timecounter.method: 0
> kern.timecounter.hardware: TSC
> kern.openfiles: 52
> kern.kq_calloutmax: 4096
> kern.ps_arg_cache_limit: 256
> kern.ps_argsopen: 1
> kern.fast_vfork: 1
> kern.randompid: 0
> kern.maxusers: 256
> kern.ps_showallprocs: 1
> kern.shutdown.poweroff_delay: 5000
> kern.shutdown.kproc_shutdown_wait: 60
> kern.sugid_coredump: 1
> kern.coredump: 1
> kern.corefile: /tmp/%P.%N.core
> kern.quantum: 100000
> kern.ccpu: 1948
> kern.fscale: 2048
> kern.devstat.numdevs: 3
> kern.devstat.generation: 3
> kern.devstat.version: 4
> kern.disks: da0
> kern.log_wakeups_per_second: 5
> kern.msgbuf:
> kern.msgbuf_clear: 0
> kern.nselcoll: 0
> kern.consmute: 0
> kern.filedelay: 30
> kern.dirdelay: 29
> kern.metadelay: 28
> kern.minvnodes: 17103
> kern.chroot_allow_open_directories: 1
> vm.loadavg: { 0.01 0.04 0.01 }
> vm.v_free_min: 1700
> vm.v_free_target: 7216
> vm.v_free_reserved: 416
> vm.v_inactive_target: 10824
> vm.v_cache_min: 7216
> vm.v_cache_max: 14432
> vm.v_pageout_free_min: 34
> vm.pageout_algorithm: 0
> vm.swap_enabled: 1
> vm.swap_async_max: 4
> vm.swap_idle_threshold1: 2
> vm.swap_idle_threshold2: 10
> vm.v_free_severe: 1058
> vm.stats.sys.v_swtch: 593028
> vm.stats.sys.v_trap: 83568
> vm.stats.sys.v_syscall: 1168881
> vm.stats.sys.v_intr: 1537519
> vm.stats.sys.v_soft: 137125
> vm.stats.vm.v_vm_faults: 84904
> vm.stats.vm.v_cow_faults: 6328
> vm.stats.vm.v_cow_optim: 0
> vm.stats.vm.v_zfod: 12331
> vm.stats.vm.v_ozfod: 12052
> vm.stats.vm.v_swapin: 0
> vm.stats.vm.v_swapout: 0
> vm.stats.vm.v_swappgsin: 0
> vm.stats.vm.v_swappgsout: 0
> vm.stats.vm.v_vnodein: 273
> vm.stats.vm.v_vnodeout: 0
> vm.stats.vm.v_vnodepgsin: 1986
> vm.stats.vm.v_vnodepgsout: 0
> vm.stats.vm.v_intrans: 0
> vm.stats.vm.v_reactivated: 119
> vm.stats.vm.v_pdwakeups: 0
> vm.stats.vm.v_pdpages: 0
> vm.stats.vm.v_dfree: 0
> vm.stats.vm.v_pfree: 15351
> vm.stats.vm.v_tfree: 20808
> vm.stats.vm.v_page_size: 4096
> vm.stats.vm.v_page_count: 257185
> vm.stats.vm.v_free_reserved: 416
> vm.stats.vm.v_free_target: 7216
> vm.stats.vm.v_free_min: 1700
> vm.stats.vm.v_free_count: 135965
> vm.stats.vm.v_wire_count: 29790
> vm.stats.vm.v_active_count: 1179
> vm.stats.vm.v_inactive_target: 10824
> vm.stats.vm.v_inactive_count: 90238
> vm.stats.vm.v_cache_count: 13
> vm.stats.vm.v_cache_min: 7216
> vm.stats.vm.v_cache_max: 14432
> vm.stats.vm.v_pageout_free_min: 34
> vm.stats.vm.v_interrupt_free_min: 2
> vm.stats.vm.v_forks: 320
> vm.stats.vm.v_vforks: 23
> vm.stats.vm.v_rforks: 0
> vm.stats.vm.v_kthreads: 4
> vm.stats.vm.v_forkpages: 20423
> vm.stats.vm.v_vforkpages: 1035
> vm.stats.vm.v_rforkpages: 0
> vm.stats.vm.v_kthreadpages: 0
> vm.stats.misc.zero_page_count: 135553
> vm.stats.misc.cnt_prezero: 203048
> vm.max_proc_mmap: 43690
> vm.max_launder: 32
> vm.pageout_stats_max: 7216
> vm.pageout_full_stats_interval: 20
> vm.pageout_stats_interval: 5
> vm.pageout_stats_free_max: 5
> vm.swap_idle_enabled: 0
> vm.defer_swapspace_pageouts: 0
> vm.disable_swapspace_pageouts: 0
> vm.zone:
> ITEM SIZE LIMIT USED FREE REQUESTS
>
> PIPE: 160, 0, 0, 102, 158
> SWAPMETA: 160, 257185, 0, 0, 0
> ripcb: 192, 65536, 0, 21, 1
> syncache: 160, 15359, 16, 444, 94100
> tcpcb: 544, 65536, 1828, 6597, 92979
> udpcb: 192, 65536, 14, 28, 71
> unpcb: 64, 0, 2, 126, 28
> socket: 192, 65536, 3503, 4966, 93080
> KNOTE: 64, 0, 0, 128, 13
> NFSNODE: 352, 0, 39781, 17, 41857
> NFSMOUNT: 544, 0, 1, 6, 1
> VNODE: 192, 0, 40016, 52, 40016
> NAMEI: 1024, 0, 0, 16, 187371
> VMSPACE: 192, 0, 34, 30, 343
> PROC: 416, 0, 38, 60, 347
> DP fakepg: 64, 0, 0, 0, 0
> PV ENTRY: 28, 1347438, 19011, 243108, 106251
> MAP ENTRY: 48, 0, 391, 162, 15581
> KMAP ENTRY: 48, 64424, 116, 97, 53824
> MAP: 108, 0, 7, 3, 7
> VM OBJECT: 96, 0, 33375, 103, 37919
> vm.zone_kmem_pages: 1538
> vm.zone_kmem_kvaspace: 150491136
> vm.zone_kern_pages: 6330
> vm.kvm_size: 1069547520
> vm.kvm_free: 213909504
> vfs.nfs.realign_test: 216199
> vfs.nfs.realign_count: 0
> vfs.nfs.bufpackets: 4
> vfs.nfs.defect: 0
> vfs.nfs.diskless_valid: 0
> vfs.nfs.diskless_rootpath:
> vfs.nfs.diskless_swappath:
> vfs.nfs.access_cache_timeout: 600
> vfs.nfs.nfsv3_commit_on_close: 0
> vfs.numdirtybuffers: 3
> vfs.lodirtybuffers: 910
> vfs.hidirtybuffers: 1820
> vfs.numfreebuffers: 7200
> vfs.lofreebuffers: 405
> vfs.hifreebuffers: 810
> vfs.runningbufspace: 0
> vfs.lorunningspace: 524288
> vfs.hirunningspace: 1048576
> vfs.maxbufspace: 118013952
> vfs.hibufspace: 117358592
> vfs.lobufspace: 117293056
> vfs.bufspace: 117293056
> vfs.maxmallocbufspace: 5867929
> vfs.bufmallocspace: 129024
> vfs.getnewbufcalls: 58745
> vfs.getnewbufrestarts: 0
> vfs.vmiodirenable: 1
> vfs.bufdefragcnt: 0
> vfs.buffreekvacnt: 0
> vfs.bufreusecnt: 7159
> vfs.cache.numneg: 118
> vfs.cache.numcache: 40114
> vfs.cache.numcalls: 760858
> vfs.cache.dothits: 112
> vfs.cache.dotdothits: 13703
> vfs.cache.numchecks: 788978
> vfs.cache.nummiss: 45111
> vfs.cache.nummisszap: 8
> vfs.cache.numposzaps: 12
> vfs.cache.numposhits: 701263
> vfs.cache.numnegzaps: 2
> vfs.cache.numneghits: 647
> vfs.cache.numcwdcalls: 32
> vfs.cache.numcwdfail1: 0
> vfs.cache.numcwdfail2: 0
> vfs.cache.numcwdfail3: 0
> vfs.cache.numcwdfail4: 0
> vfs.cache.numcwdfound: 32
> vfs.cache.numfullpathcalls: 0
> vfs.cache.numfullpathfail1: 0
> vfs.cache.numfullpathfail2: 0
> vfs.cache.numfullpathfail3: 0
> vfs.cache.numfullpathfail4: 0
> vfs.cache.numfullpathfound: 0
> vfs.write_behind: 1
> vfs.reassignbufcalls: 493
> vfs.reassignbufloops: 0
> vfs.reassignbufsortgood: 174
> vfs.reassignbufsortbad: 75
> vfs.reassignbufmethod: 1
> vfs.nameileafonly: 0
> vfs.timestamp_precision: 0
> vfs.usermount: 0
> vfs.ffs.doreallocblks: 1
> vfs.ffs.doasyncfree: 1
> net.local.stream.sendspace: 8192
> net.local.stream.recvspace: 8192
> net.local.dgram.maxdgram: 2048
> net.local.dgram.recvspace: 4096
> net.local.inflight: 0
> net.inet.ip.portrange.lowfirst: 1023
> net.inet.ip.portrange.lowlast: 600
> net.inet.ip.portrange.first: 1024
> net.inet.ip.portrange.last: 8000
> net.inet.ip.portrange.hifirst: 49152
> net.inet.ip.portrange.hilast: 65535
> net.inet.ip.forwarding: 0
> net.inet.ip.redirect: 1
> net.inet.ip.ttl: 64
> net.inet.ip.rtexpire: 2400
> net.inet.ip.rtminexpire: 10
> net.inet.ip.rtmaxcache: 128
> net.inet.ip.sourceroute: 0
> net.inet.ip.intr_queue_maxlen: 100
> net.inet.ip.intr_queue_drops: 0
> net.inet.ip.accept_sourceroute: 0
> net.inet.ip.fastforwarding: 0
> net.inet.ip.keepfaith: 0
> net.inet.ip.subnets_are_local: 0
> net.inet.ip.maxfragpackets: 16384
> net.inet.ip.check_interface: 0
> net.inet.icmp.maskrepl: 0
> net.inet.icmp.icmplim: 4000
> net.inet.icmp.drop_redirect: 0
> net.inet.icmp.log_redirect: 0
> net.inet.icmp.bmcastecho: 0
> net.inet.tcp.rfc1323: 1
> net.inet.tcp.rfc1644: 1
> net.inet.tcp.mssdflt: 1460
> net.inet.tcp.keepidle: 7200000
> net.inet.tcp.keepintvl: 75000
> net.inet.tcp.sendspace: 131072
> net.inet.tcp.recvspace: 4096
> net.inet.tcp.keepinit: 75000
> net.inet.tcp.delacktime: 100
> net.inet.tcp.log_in_vain: 0
> net.inet.tcp.blackhole: 0
> net.inet.tcp.delayed_ack: 1
> net.inet.tcp.path_mtu_discovery: 1
> net.inet.tcp.slowstart_flightsize: 1
> net.inet.tcp.local_slowstart_flightsize: 4
> net.inet.tcp.newreno: 1
> net.inet.tcp.tcbhashsize: 512
> net.inet.tcp.do_tcpdrain: 1
> net.inet.tcp.pcbcount: 1827
> net.inet.tcp.icmp_may_rst: 1
> net.inet.tcp.strict_rfc1948: 0
> net.inet.tcp.isn_reseed_interval: 0
> net.inet.tcp.syncookies: 1
> net.inet.tcp.syncache.bucketlimit: 30
> net.inet.tcp.syncache.cachelimit: 15359
> net.inet.tcp.syncache.count: 16
> net.inet.tcp.syncache.hashsize: 512
> net.inet.tcp.syncache.rexmtlimit: 3
> net.inet.tcp.msl: 20000
> net.inet.tcp.always_keepalive: 1
> net.inet.udp.checksum: 1
> net.inet.udp.maxdgram: 9216
> net.inet.udp.recvspace: 41600
> net.inet.udp.log_in_vain: 0
> net.inet.udp.blackhole: 0
> net.inet.accf.unloadable: 0
> net.inet.accf.http.parsehttpversion: 0
> net.inet.raw.maxdgram: 8192
> net.inet.raw.recvspace: 8192
> net.link.generic.system.ifcount: 3
> net.link.ether.inet.prune_intvl: 300
> net.link.ether.inet.max_age: 1200
> net.link.ether.inet.host_down_time: 20
> net.link.ether.inet.maxtries: 5
> net.link.ether.inet.useloopback: 1
> net.link.ether.inet.proxyall: 0
> net.link.ether.inet.log_arp_wrong_iface: 1
> debug.elf_trace: 0
> debug.boothowto: -2147479552
> debug.free_devt: 0
> debug.fdexpand: 4
> debug.sizeof.vnode: 168
> debug.sizeof.proc: 408
> debug.sizeof.specinfo: 68
> debug.sizeof.disklabel: 276
> debug.sizeof.diskslices: 1820
> debug.sizeof.disk: 304
> debug.ttydebug: 0
> debug.nchash: 131071
> debug.ncnegfactor: 16
> debug.numneg: 118
> debug.numcache: 40114
> debug.vfscache: 1
> debug.vnsize: 168
> debug.ncsize: 36
> debug.disablecwd: 0
> debug.disablefullpath: 0
> debug.numvnodes: 40016
> debug.wantfreevnodes: 25
> debug.freevnodes: 26
> debug.rush_requests: 0
> debug.max_softdeps: 547296
> debug.tickdelay: 2
> debug.worklist_push: 0
> debug.blk_limit_push: 0
> debug.ino_limit_push: 0
> debug.blk_limit_hit: 0
> debug.ino_limit_hit: 0
> debug.sync_limit_hit: 0
> debug.indir_blk_ptrs: 0
> debug.inode_bitmap: 0
> debug.direct_blk_ptrs: 0
> debug.dir_entry: 0
> debug.dircheck: 0
> hw.machine: i386
> hw.model: Pentium III/Pentium III Xeon/Celeron
> hw.ncpu: 1
> hw.byteorder: 1234
> hw.physmem: 1071288320
> hw.usermem: 949264384
> hw.pagesize: 4096
> hw.floatingpoint: 1
> hw.machine_arch: i386
> hw.instruction_sse: 0
> hw.availpages: 261379
> hw.fxp0.int_delay: 1000
> hw.fxp0.bundle_max: 6
> hw.fxp1.int_delay: 1000
> hw.fxp1.bundle_max: 6
> machdep.consdev: { major = 12, minor = 255 }
> machdep.adjkerntz: 0
> machdep.disable_rtc_set: 0
> machdep.wall_cmos_clock: 0
> machdep.do_dump: 1
> machdep.enable_panic_key: 0
> machdep.ispc98: 0
> machdep.msgbuf:
> machdep.msgbuf_clear: 0
> machdep.panic_on_nmi: 1
> machdep.i8254_freq: 1193182
> machdep.tsc_freq: 866427923
> user.cs_path: /usr/bin:/bin:/usr/sbin:/sbin:
> user.bc_base_max: 99
> user.bc_dim_max: 2048
> user.bc_scale_max: 99
> user.bc_string_max: 1000
> user.coll_weights_max: 0
> user.expr_nest_max: 32
> user.line_max: 2048
> user.re_dup_max: 255
> user.posix2_version: 199212
> user.posix2_c_bind: 0
> user.posix2_c_dev: 0
> user.posix2_char_term: 0
> user.posix2_fort_dev: 0
> user.posix2_fort_run: 0
> user.posix2_localedef: 0
> user.posix2_sw_dev: 0
> user.posix2_upe: 0
> user.stream_max: 20
> user.tzname_max: 255
> p1003_1b.asynchronous_io: 0
> p1003_1b.mapped_files: 0
> p1003_1b.memlock: 0
> p1003_1b.memlock_range: 0
> p1003_1b.memory_protection: 0
> p1003_1b.message_passing: 0
> p1003_1b.prioritized_io: 0
> p1003_1b.priority_scheduling: 1
> p1003_1b.realtime_signals: 0
> p1003_1b.semaphores: 0
> p1003_1b.fsync: 0
> p1003_1b.shared_memory_objects: 0
> p1003_1b.synchronized_io: 0
> p1003_1b.timers: 0
> p1003_1b.aio_listio_max: 0
> p1003_1b.aio_max: 0
> p1003_1b.aio_prio_delta_max: 0
> p1003_1b.delaytimer_max: 0
> p1003_1b.mq_open_max: 0
> p1003_1b.pagesize: 4096
> p1003_1b.rtsig_max: 0
> p1003_1b.sem_nsems_max: 0
> p1003_1b.sem_value_max: 0
> p1003_1b.sigqueue_max: 0
> p1003_1b.timer_max: 0
> jail.set_hostname_allowed: 1
> jail.socket_unixiproute_only: 1
> jail.sysvipc_allowed: 0
>
> If anyone has any ideas, please suggest.
>
> --Isprime Support
>
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-stable" in the body of the message
