Hi,
I am passing my finding around for the configuration of sysctl.conf to
remove bottleneck I found in httpd as I couldn't get more then 300 httpd
process without crapping out badly and above that, the server simply got
out of wack.
All is default install and the tests are done with a server that is an
old one. dmesg at the end in case you are interested. This is on OpenBSD
4.0 and I pick that server just to see what's possible as it's not
really a very powerful one.
You can also see the iostat output and the vmstat as well with the
changes in place.
You sure can see a few page fault as I am really pushing the server
much, but even then I get decent results and the bottleneck was remove,
even with 2000 parallel connections. In that case I had to use two
different clients as the http_load only support up to 1021 parallel
connections, so to test pass that, I use more then one clients to push
the server more.
But in all, the results are much better then a few days ago and now
looks like we get more for the buck and adding more powerful hardware
will be use better now instead of suffering the same limitations.
I put also the value changed in sysctl.conf to come to this final setup.
I am not saying the value are the best possible choice, but they work
well in the test situation and there is many as you will see. Some are
very surprising to me, like the change in net.inet.ip.portfirst. Yes I
know, but if I leave it as default, then I can't get full success in the
test below and get time out, some errors and efficiency is not as good.
May be that's because of the random ports range calculations, I can't
say, but in any case, the effect is there and tested.
I try to stay safe in my choices and comments are welcome, but I have to
point out as well that ALL the values below needs to be changes to that
new value to get working well. If even only one of them is not at the
level below, the results in the tests start to be affected pretty bad at
times.
So, not only one value needs to be changed or address the issues, but
ALL of them below.
I am still working on finding may be more restrictive value to keep the
system as stable and safe and close to the default as possible, but
below is a very good setup in y tests and all the results are below as well.
As for the value in httpd.conf, they are still in progress to make them
more normal, but for this test they are:
Timeout 300
KeepAlive On
MaxKeepAliveRequests 0 (shouldn't stay like this as limits needs to be
in place)
KeepAliveTimeout 5
MinSpareServers 40
MaxSpareServers 80
StartServers 40
MaxClients 2048
MaxRequestsPerChild 0
Also, the httpd use .so module like php and is not compile statically.
For the value above, I think a more reasonable (still in progress as
well) would be for a very busy server:
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
MinSpareServers 50
MaxSpareServers 100
StartServers 75
MaxClients 768
MaxRequestsPerChild 0
However, I am not settle on them fully yet. I send an earlier email with
explication for why some value should be pick.
http://marc.info/?l=openbsd-misc&m=117874246431437&w=2
Any comments on any parts or caution I have overlooked?
Thanks and hope this help some others that may suffer from the same
problem I did.
Daniel
===========================
sysctl.conf changes.
kern.seminfo.semmni=1024
kern.seminfo.semmns=4096
kern.shminfo.shmall=16384
kern.maxclusters=12000
kern.maxproc=2048 # Increase for the process limits.
kern.maxfiles=5000
kern.shminfo.shmmax=67108864
kern.somaxconn=2048
net.bpf.bufsize=524288
net.inet.ip.maxqueue=1278
net.inet.ip.portfirst=32768
net.inet.ip.redirect=0
net.inet.tcp.keepinittime=10
net.inet.tcp.keepidle=30
net.inet.tcp.keepintvl=30
net.inet.tcp.mssdflt=1452
net.inet.tcp.recvspace=65535
net.inet.tcp.rstppslimit=400
net.inet.tcp.sendspace=65535
net.inet.tcp.synbucketlimit=420
net.inet.tcp.syncachelimit=20510
===========================
Test with multiple parallel connections, from 10 to 1000. As expected,
the results gets better as we go and I was able to go up to 2000, but I
limit the server at 2048 in the recompile version. At 2000, I get close
to 2x the delay, meaning it's start to go back up before that, but still
get full completed without errors in less then the time out of 30
seconds, witch I couldn't do before at 300 parallel connections anyway.
# http_load -parallel 10 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 10 max parallel, 1.9647e+07 bytes, in 19.8742 seconds
13098 mean bytes/connection
75.4747 fetches/sec, 988568 bytes/sec
msecs/connect: 84.6428 mean, 6003.03 max, 0.347 min
msecs/first-response: 17.6985 mean, 1698.64 max, 3.236 min
HTTP response codes:
code 200 -- 1500
# http_load -parallel 20 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 20 max parallel, 1.9647e+07 bytes, in 20.824 seconds
13098 mean bytes/connection
72.0324 fetches/sec, 943480 bytes/sec
msecs/connect: 196.569 mean, 6003.31 max, 0.345 min
msecs/first-response: 25.5585 mean, 803.003 max, 3.202 min
HTTP response codes:
code 200 -- 1500
# http_load -parallel 50 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 50 max parallel, 1.9647e+07 bytes, in 14.8809 seconds
13098 mean bytes/connection
100.801 fetches/sec, 1.32029e+06 bytes/sec
msecs/connect: 284.599 mean, 6003.93 max, 0.393 min
msecs/first-response: 66.0069 mean, 2194.22 max, 3.209 min
HTTP response codes:
code 200 -- 1500
# http_load -parallel 100 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 100 max parallel, 1.9647e+07 bytes, in 13.1644 seconds
13098 mean bytes/connection
113.944 fetches/sec, 1.49244e+06 bytes/sec
msecs/connect: 304.66 mean, 6003.6 max, 0.424 min
msecs/first-response: 259.759 mean, 4096.73 max, 3.236 min
HTTP response codes:
code 200 -- 1500
# http_load -parallel 250 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 250 max parallel, 1.9647e+07 bytes, in 12.4591 seconds
13098 mean bytes/connection
120.394 fetches/sec, 1.57692e+06 bytes/sec
msecs/connect: 300.82 mean, 6004.53 max, 0.401 min
msecs/first-response: 876.977 mean, 4570.27 max, 3.34 min
HTTP response codes:
code 200 -- 1500
# http_load -parallel 500 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 500 max parallel, 1.9647e+07 bytes, in 11.2113 seconds
13098 mean bytes/connection
133.793 fetches/sec, 1.75242e+06 bytes/sec
msecs/connect: 221.101 mean, 6004.34 max, 0.437 min
msecs/first-response: 1850.79 mean, 5473.84 max, 3.376 min
HTTP response codes:
code 200 -- 1500
# http_load -parallel 1000 -fetches 1500 -timeout 30 /tmp/test
1500 fetches, 1000 max parallel, 1.9647e+07 bytes, in 8.75952 seconds
13098 mean bytes/connection
171.242 fetches/sec, 2.24293e+06 bytes/sec
msecs/connect: 1877.58 mean, 6006.61 max, 0.404 min
msecs/first-response: 1686.37 mean, 4582.26 max, 3.488 min
HTTP response codes:
code 200 -- 1500
#
===================================
iostat 1
# iostat 1
tty wd0 wd1 cd0 cpu
tin tout KB/t t/s MB/s KB/t t/s MB/s KB/t t/s MB/s us ni sy in id
0 1 7.46 0 0.00 10.87 0 0.00 0.00 0 0.00 0 0 1 0 99
0 220 8.00 1 0.01 0.00 0 0.00 0.00 0 0.00 0 0 3 0 97
0 73 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
0 73 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
0 73 8.00 1 0.01 0.00 0 0.00 0.00 0 0.00 0 0 0 1 99
0 73 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 0 0100
0 73 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 45 0 16 8 32
0 54 1.50 1 0.00 0.00 0 0.00 0.00 0 0.00 57 0 29 13 0
0 53 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 58 0 31 11 0
0 53 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 69 0 21 11 0
0 45 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 26 0 15 7 53
0 49 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 68 0 23 9 0
0 65 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 14 0 8 5 73
0 73 8.00 2 0.02 0.00 0 0.00 0.00 0 0.00 0 0 2 0 98
0 73 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0 0 2 0 98
0 73 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 0 1 0 98
===================================
vmstat 1
# vmstat 1
procs memory page disks traps cpu
r b w avm fre flt re pi po fr sr wd0 wd1 int sys cs
us sy id
0 2 0 709841760620 28 0 0 0 0 0 0 0 292 162 30
0 1 99
0 2 0 709921760612 26 0 0 0 0 0 0 0 257 67 9
0 0 100
0 2 0 709921760612 9 0 0 0 0 0 0 0 260 25 8
0 0 100
81 2 0 715041759868 628 0 0 0 0 0 0 0 4367 19628 372
50 43 8
81 2 0 719441758728 1672 0 0 0 0 0 0 0 4617 31535 570
66 34 0
83 2 0 730041756240 2819 0 0 0 0 0 0 0 4346 32988 590
64 36 0
74 2 0 760281749924 5735 0 0 0 0 0 0 0 3962 32892 628
63 37 0
114 2 0 811161739364 9639 0 0 0 0 0 0 0 2884 19747 425
31 24 45
34 2 0 860161727552 9831 0 0 0 0 0 0 0 4382 35238 767
60 40 0
0 2 0 861521727264 1376 0 0 0 0 0 0 0 599 2168 272
1 10 89
0 2 0 860761727524 539 0 0 0 0 0 0 0 260 124 11
2 0 98
0 2 0 860001727784 539 0 0 0 0 0 0 0 281 137 13
1 3 96
0 2 0 859241728044 539 0 0 0 0 0 0 0 258 124 12
1 2 97
===================================
dmesg
OpenBSD 4.0 (GENERIC) #1107: Sat Sep 16 19:15:58 MDT 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 845 MHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,SER,MMX,FXSR,SSE
real mem = 2138664960 (2088540K)
avail mem = 1942753280 (1897220K)
using 4256 buffers containing 107036672 bytes (104528K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 07/15/95, BIOS32 rev. 0 @
0xfdb70, SMBIOS rev. 2.1 @ 0xf0770 (32 entries)
bios0: GDA Intel 440GX
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI BIOS has 8 Interrupt Routing table entries
pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371AB PIIX4 ISA"
rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc0000/0x9600
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82440BX AGP" rev 0x00
ppb0 at pci0 dev 1 function 0 "Intel 82440BX AGP" rev 0x00
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "Chips and Technologies 69000" rev 0x64
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x02
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <QUANTUM FIREBALLP LM30>
wd0: 16-sector PIO, LBA, 28629MB, 58633344 sectors
wd1 at pciide0 channel 0 drive 1: <QUANTUM FIREBALLP LM30>
wd1: 16-sector PIO, LBA, 28629MB, 58633344 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
wd1(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
atapiscsi0 at pciide0 channel 1 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TEAC, CD-224E, 1.5A> SCSI0 5/cdrom removable
cd0(pciide0:1:1): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 7 function 2 "Intel 82371AB USB" rev 0x01pci_intr_map:
no mapping for pin D
: couldn't map interrupt
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x02: polling
iic0 at piixpm0
admtemp0 at iic0 addr 0x18: max1617
admtemp1 at iic0 addr 0x1a: max1617
"unknown" at iic0 addr 0x2d not configured
maxtmp0 at iic0 addr 0x4c: max6657
maxtmp1 at iic0 addr 0x4e: max6657
fxp0 at pci0 dev 17 function 0 "Intel 8255x" rev 0x08, i82559: irq 11,
address 00:d0:a8:00:60:18
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
fxp1 at pci0 dev 18 function 0 "Intel 8255x" rev 0x08, i82559: irq 10,
address 00:d0:a8:00:60:19
inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4
ppb1 at pci0 dev 20 function 0 "DEC 21152 PCI-PCI" rev 0x03
pci2 at ppb1 bus 2
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
biomask f3fd netmask fffd ttymask ffff
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
dkcsum: wd0 matches BIOS drive 0x80
dkcsum: wd1 matches BIOS drive 0x81
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
