A customer of a customer needed to expand the size of the filesystem on a
webserver they run, and since it was still running 4.11, they decided to
go ahead and upgrade to 7.1.
Unfortunately, they have seen the performance decrease significantly. Now,
under 7.1, numerous perl processes will queue up (from their web
applications), the load average will be in the 40s to 90s, and according
to top they perl scripts are commonly in the "ufs" state, which I assume
means stuck waiting for read or write responses.
I have also experienced pretty abysmal read performance from the array,
using dd...we're talking sub 1MBps. So, my assumption is that something is
wrong on the filesystem layer.
When I look at the CPU utilization in top, I see that 80-90% is constantly
used by "system".
However, when I look at iostat, I see very low numbers for the raid, in
fact across the board.
It's a Dell server with a Perc4/Di RAID controller, which uses the amr
driver. They upgraded to larger, 15k rpm ultra 320 disks (from smaller 15k
rpm ultra 320 disks). They made a backup of their web root, which is a
seperate partition, using dump to a temporary drive, then swapped in the
new disks, installed FreeBSD 7.1, and restored their webroot to a
partition on the new array.
Does anybody have any insight into what could be going on here?
dmesg is below.
Thanks!
Andy
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-RELEASE #0: Thu Jan 1 14:37:25 UTC 2009
r...@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3056.81-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff
Features2=0x4400
Logical CPUs per core: 2
real memory = 4160552960 (3967 MB)
avail memory = 4069429248 (3880 MB)
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic2: Changing APIC ID to 10
ioapic3: Changing APIC ID to 11
ioapic4: Changing APIC ID to 12
ioapic0 irqs 0-23 on motherboard
ioapic1 irqs 24-47 on motherboard
ioapic2 irqs 72-95 on motherboard
ioapic3 irqs 120-143 on motherboard
ioapic4 irqs 144-167 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: port 0xcf8-0xcff on acpi0
pci0: on pcib0
pcib1: at device 2.0 on pci0
pci1: on pcib1
pcib2: at device 29.0 on pci1
pci2: on pcib2
pcib3: at device 31.0 on pci1
pci3: on pcib3
em0: port 0xece0-0xecff mem
0xfe8e-0xfe8f,0xfe8c-0xfe8d irq 28 at device 1.0 on pci3
em0: [FILTER]
em0: Ethernet address: 00:0f:1f:69:e3:7d
pcib4: at device 3.0 on pci0
pci4: on pcib4
pcib5: at device 29.0 on pci4
pci5: on pcib5
pcib6: at device 31.0 on pci4
pci6: on pcib6
pcib7: at device 4.0 on pci0
pci7: on pcib7
pcib8: at device 29.0 on pci7
pci8: on pcib8
amr0: mem 0xfebf-0xfebf irq 120 at device
8.0 on pci8
amr0: [ITHREAD]
amr0: delete logical drives supported by controller
amr0: Firmware 2.48, BIOS 1.06, 128MB RAM
pcib9: at device 31.0 on pci7
pci10: on pcib9
uhci0: port 0xcce0-0xccff
irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: on uhci0
usb0: USB revision 1.0
uhub0: on usb0
uhub0: 2 ports with 2 removable, self powered
pcib10: at device 30.0 on pci0
pci11: on pcib10
vgapci0: port 0xdc00-0xdcff mem
0xfc00-0xfcff,0xfdeff000-0xfdef at device 4.0 on pci11
isab0: at device 31.0 on pci0
isa0: on isab0
atapci0: port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: on atapci0
ata0: [ITHREAD]
ata1: on atapci0
ata1: [ITHREAD]
fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: port 0x60,0x64 irq 1 on acpi0
atkbd0: irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse Explorer, device ID 4
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
cpu0: on acpi0
p4tcc0: on cpu0
cpu1: on acpi0
p4tcc1: on cpu1
cpu2: on acpi0
p4tcc2: on cpu2
cpu3: on acpi0
p4tcc3: on cpu3
pmtimer0 on isa0
orm0: at iomem 0xc-0xc7fff,0xec000-0xe pnpid
ORM on isa0
ppc0: at port 0x378-0x37f
