Hello everyone,
I running into some problems with a Cyrus Imap server. A year ago I
asked about sizing the server and it was defined as "overkill" (Dell
1850, Perc 4e/DC, 4 300GB 10krpm disks in Raid 10) for 300 users, but
now I'm suffering of really high loads and performance problems. I
think it's due I/O contention (more than a thousand t/s in iostat),
but don't know why Cyrus is so intensive in I/O that this hardware
can't handle it. Memory consumption is also very high.
When I started with ~60 users everything was ok, but after surpasing
the one hundred, performance problems arised. I added a imap proxy
which sited between the server and the webmail(Squirrel) and improved
overall performance, but now we have 215 users and the server has
severe problems during peak hours, when load can reach values of 12 or
more. Users perceive degraded response times.
Top usually lota of imap processes waiting for getblk, biowait, lockf,
or select. Vmstat also shows always proceses blocked waiting for I/O.
I had to tweak default kernel values because server crashed when
importing old mailboxes:
maxusers 64 # estimated number of users
option NKMEMPAGES_MAX=65535
option BUFCACHEPERCENT=15
Cyrus was compiled from ports, with default options. The server is
running Sendmail and a lightly loaded openldap.
I don't know where to look at or what to tweak. Any ideas will be welcome.
Best regards.
Dmesg and other relevant data (non peak hours).
load averages: 10.36, 6.97, 5.79
17:59:50
259 processes: 258 idle, 1 on processor
CPU states: 3.7% user, 0.0% nice, 20.6% system, 1.9% interrupt, 73.8% idle
Memory: Real: 1135M/1702M act/tot Free: 312M Swap: 47M/2196M used/tot
PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
9503 _cyrus 2 0 2716K 4392K sleep select 0:08 1.12% imapd
16618 _cyrus -5 0 3976K 5728K sleep biowai 0:08 1.03% imapd
14158 _cyrus 2 0 3544K 5568K sleep select 0:01 0.68% imapd
23082 _cyrus -5 0 11M 11M sleep getblk 0:06 0.54% imapd
4397 _cyrus 2 0 1664K 3396K sleep select 0:00 0.54% imapd
831 _cyrus -5 0 14M 16M sleep getblk 0:01 0.34% imapd
30096 _cyrus 2 0 6136K 7464K sleep select 0:03 0.29% imapd
10307 _cyrus -5 0 5572K 5908K sleep getblk 0:00 0.29% imapd
28758 _cyrus -5 0 7664K 9412K sleep biowai 0:02 0.20% imapd
27091 _cyrus -5 0 10M 12M sleep getblk 0:02 0.20% imapd
15191 _cyrus 2 0 1740K 3664K sleep select 0:00 0.20% imapd
17387 _cyrus 2 0 2216K 4056K sleep select 0:02 0.15% imapd
25614 _cyrus 2 0 2056K 3952K sleep select 0:02 0.15% imapd
# iostat 5 5
tty sd0 sd1 cd0
fd0 cpu
tin tout KB/t t/s MB/s KB/t t/s MB/s KB/t t/s MB/s KB/t t/s
MB/s us ni sy in id
0 8 10.77 18 0.19 14.30 242 3.38 0.00 0 0.00 0.00 0
0.00 18 0 5 1 77
0 54 9.93 34 0.33 15.00 956 14.00 0.00 0 0.00 0.00 0
0.00 3 0 13 0 84
0 18 12.07 80 0.95 14.33 615 8.61 0.00 0 0.00 0.00 0
0.00 7 0 15 1 77
0 18 8.68 9 0.08 14.03 575 7.88 0.00 0 0.00 0.00 0
0.00 1 0 6 1 92
0 18 10.21 30 0.30 13.26 709 9.19 0.00 0 0.00 0.00 0
0.00 1 0 9 2 88
# vmstat 5 5
procs memory page disks traps cpu
r b w avm fre flt re pi po fr sr sd0 sd1 int sys cs us sy id
7 2 01206040 321392 1719 0 0 11 0 174 12 430 573 4294967005
446 18 6 77
1 7 01200372 329816 7117 0 0 0 0 0 28 1497 1488 8354 1469 11 17 72
0 2 01198584 331560 4741 0 0 0 0 0 15 1770 1390 5770 1202 3 14 83
0 2 01189932 341820 2549 0 0 0 0 0 37 778 800 2981 592 2 8 90
1 5 01196632 334756 5348 0 0 0 0 0 56 1622 1455 6145 1247 3 18 78
/etc/sysctl.conf :
kern.maxproc=1024
kern.maxfiles=8000
net.inet.tcp.sendspace=65535
net.inet.tcp.recvspace=65535
# mount
/dev/sd0a on / type ffs (local)
/dev/sd0i on /home type ffs (local, nodev, softdep)
/dev/sd0d on /tmp type ffs (local, nodev, nosuid)
/dev/sd0f on /usr type ffs (local, nodev)
/dev/sd0e on /var type ffs (local, nodev, nosuid, softdep)
/dev/sd0h on /var/spool type ffs (local, noatime, nodev, nosuid, softdep)
/dev/sd1a on /das0 type ffs (local, noatime, nodev, nosuid, softdep)
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 500M 99.9M 375M 21% /
/dev/sd0i 50.7G 42.4G 5.8G 88% /home
/dev/sd0d 124M 16.0K 117M 0% /tmp
/dev/sd0f 7.9G 3.3G 4.2G 44% /usr
/dev/sd0e 2.0G 344M 1.5G 18% /var
/dev/sd0h 3.9G 55.7M 3.7G 1% /var/spool
/dev/sd1a 550G 78.3G 444G 15% /das0
# cat /etc/cyrus.conf
START {
# do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# this is only necessary if using idled for IMAP IDLE
idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
# add or remove based on preferences
imap cmd="imapd -U 30" listen="imap" prefork=10
# imaps cmd="imapd -s" listen="imaps" prefork=0
pop3 cmd="pop3d" listen="pop3" prefork=1
# pop3s cmd="pop3d -s" listen="pop3s" prefork=0
sieve cmd="timsieved" listen="sieve" prefork=1
# these are only necessary if receiving/exporting usenet via NNTP
# nntp cmd="nntpd" listen="nntp" prefork=0
# nntps cmd="nntpd -s" listen="nntps" prefork=0
# at least one LMTP is required for delivery
# lmtp cmd="lmtpd" listen="lmtp" prefork=0
#lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=1
maxchild=40
lmtpunix cmd="lmtpd -a" listen="/var/imap/socket/lmtp"
prefork=5 maxchild=10
# this is only necessary if using notifications
notify cmd="notifyd" listen="/var/imap/socket/notify"
proto="udp" prefork=1
}
EVENTS {
# this is required
checkpoint cmd="ctl_cyrusdb -c" period=30
# this is only necessary if using duplicate delivery suppression,
# Sieve or NNTP
delprune cmd="cyr_expire -E 3" at=0400
squatter cmd="squatter -s -r user" at=0100
# this is only necessary if caching TLS sessions
# tlsprune cmd="tls_prune" at=0400
}
OpenBSD 3.9-stable (GENERIC) #3: Sat Aug 26 23:30:07 CEST 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.20 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
real mem = 2146807808 (2096492K)
avail mem = 1779486720 (1737780K)
using 4278 buffers containing 280363008 bytes (273792K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 01/09/06, BIOS32 rev. 0 @ 0xffe90
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb260/304 (17 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC" rev 0x00)
pcibios0: PCI bus #11 is the last bus
bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x1000 0xcc000/0x1000
0xcd000/0x2200 0xcf800/0x2200 0xec000/0x4000!
ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca8/8 spacing 4
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x09
ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x09
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x06
pci2 at ppb1 bus 2
ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: irq 7 Dell 16c 32b
ami0: FW 521X, BIOS vH430, 256MB RAM
ami0: 1 channels, 0 FC loops, 1 logical drives
scsibus0 at ami0: 40 targets
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 69880MB, 69880 cyl, 64 head, 32 sec, 512 bytes/sec, 143114240 sec total
scsibus1 at ami0: 16 targets
safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x2 SCSI BP, 1.0> SCSI2
3/processor fixed
ppb2 at pci1 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x06
pci3 at ppb2 bus 3
ppb3 at pci0 dev 4 function 0 "Intel MCH PCIE" rev 0x09
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x07
pci5 at ppb4 bus 5
ami1 at pci5 dev 14 function 0 "Symbios Logic MegaRAID 320-2E" rev
0x07: irq 7 Dell 2 32b
ami1: FW 521X, BIOS vH430, 128MB RAM
ami1: 2 channels, 0 FC loops, 1 logical drives
scsibus2 at ami1: 40 targets
sd1 at scsibus2 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd1: 572160MB, 572160 cyl, 64 head, 32 sec, 512 bytes/sec, 1171783680 sec total
scsibus3 at ami1: 16 targets
ses0 at scsibus3 targ 6 lun 0: <DELL, PV22XS, E.19> SCSI3 3/processor fixed
scsibus4 at ami1: 16 targets
ppb5 at pci4 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x07
pci6 at ppb5 bus 6
ppb6 at pci0 dev 5 function 0 "Intel MCH PCIE" rev 0x09
pci7 at ppb6 bus 7
ppb7 at pci7 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci8 at ppb7 bus 8
em0 at pci8 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
irq 11, address 00:13:72:5c:87:e9
ppb8 at pci7 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci9 at ppb8 bus 9
em1 at pci9 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
irq 3, address 00:13:72:5c:87:ea
ppb9 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x09
pci10 at ppb9 bus 10
uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: irq 11
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: irq 7
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: irq 5
usb3 at ehci0: USB revision 2.0
uhub3 at usb3
uhub3: Intel EHCI root hub, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
ppb10 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
pci11 at ppb10 bus 11
vga1 at pci11 dev 13 function 0 "ATI Radeon VE QY" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
DMA, channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus5 at atapiscsi0: 2 targets
cd0 at scsibus5 targ 0 lun 0: <TEAC, CD-ROM CD-224E-N, 3.AB> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ffe5 netmask ffed ttymask ffef
pctr: user-level cycle counter enabled
uhub4 at uhub3 port 3
uhub4: Dell product 0xa001, rev 2.00/0.00, addr 2
uhub4: 2 ports with 2 removable, self powered, multiple transaction translators
dkcsum: sd0 matches BIOS drive 0x80
dkcsum: sd1 matches BIOS drive 0x81
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
WARNING: / was not properly unmounted
arplookup: unable to enter address for 192.168.10.229
arplookup: unable to enter address for 192.168.10.243
uvm_mapent_alloc: out of static map entries
uvm_mapent_alloc: out of static map entries
uvm_mapent_alloc: out of static map entries
uvm_mapent_alloc: out of static map entries
