Hi Folks,
using a simple test program to write sequential blocks to a file, optionally
opening with O_SYNC, I've tested write performance to a MegaRAID logical
drive consisting of a RAID-5 set of 4 72GB HP Ultra320 disks and to a RAID-0
drive consisting of a single 300GB HP Ultra320 disk. The controller
has 128MB of cache, but I do not have a battery, so cache write policy
is write-through. The kernel is the "bsd", and not "bsd.mp", but it makes
no difference. I've tried both.
All tests performed with O_SYNC, to avoid bufcache interaction.
The performance, at least to my perhaps naive eyes, seems abysmal. I'm getting
1.2MB/sec on the RAID-5 logical drive.
It doesn't matter what blocksize I choose, 32k, 64k, 128k, 256k. The chunksize
on the logical drive is the default, 64k.
On the single HP 300GB Ultra320 RAID-0 drive write performance is also bad:
1.7MB/sec.
The systems is (in case you hadn't already guessed) a Compaq ProLiant DL380-G2
with 1GB main memory and dual 1266MHz CPUs. I will attach dmesg, my test
program, and the output of "bioctl -iv ami0" below.
I can't imagine the MegaRAID controller is this slow. Any suggestions?
Rob Urban
dmesg:
---------------------------------------------------------------
OpenBSD 3.9 (GENERIC) #617: Thu Mar 2 02:26:48 MST 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) III CPU family 1266MHz ("GenuineIntel" 686-class)
1.27 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,M
MX,FXSR,SSE
real mem = 1073307648 (1048152K)
avail mem = 972660736 (949864K)
using 4278 buffers containing 53768192 bytes (52508K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf0000
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x2000
pcibios0: PCI BIOS has 9 Interrupt Routing table entries
pcibios0: PCI Interrupt Router at 000:15:0 ("ServerWorks OSB4" rev 0x00)
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x2600 0xee000/0x2000!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20HE Host" rev 0x23
pci1 at pchb0 bus 1
ppb0 at pci1 dev 3 function 0 "Intel i960 RP PCI-PCI" rev 0x05
pci2 at ppb0 bus 2
vga1 at pci2 dev 0 function 0 "ATI Mach64 GV" rev 0x7a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Intel 80960RP ATU" rev 0x05 at pci1 dev 3 function 1 not configured
pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20HE Host" rev 0x01
pchb2 at pci0 dev 0 function 2 "ServerWorks CNB20HE Host" rev 0x01
pchb3 at pci0 dev 0 function 3 "ServerWorks CNB20HE Host" rev 0x01
pci3 at pchb3 bus 7
ami0 at pci3 dev 4 function 0 "Symbios Logic MegaRAID" rev 0x01: irq 3 Dell
518 64b/lhc
ami0: FW 351X, BIOS v1.10, 128MB RAM
ami0: 2 channels, 0 FC loops, 2 logical drives
scsibus0 at ami0: 40 targets
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 208110MB, 208110 cyl, 64 head, 32 sec, 512 bytes/sec, 426209280 sec total
sd1 at scsibus0 targ 1 lun 0: <AMI, Host drive #01, > SCSI2 0/direct fixed
sd1: 286080MB, 286080 cyl, 64 head, 32 sec, 512 bytes/sec, 585891840 sec total
scsibus1 at ami0: 16 targets
uk0 at scsibus1 targ 15 lun 0: <COMPAQ, PROLIANT 4L6I, 1.84> SCSI2 3/processor
fixed
uk0: unknown device
scsibus2 at ami0: 16 targets
"Compaq PCI Hotplug" rev 0x12 at pci3 dev 7 function 0 not configured
fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x08, i82559: irq 5, address
00:08:02:8a:4b:fc
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
fxp1 at pci0 dev 4 function 0 "Intel 8255x" rev 0x08, i82559: irq 7, address
00:08:02:8a:4b:fb
inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4
"Compaq Netelligent ASMC" rev 0x00 at pci0 dev 6 function 0 not configured
piixpm0 at pci0 dev 15 function 0 "ServerWorks OSB4" rev 0x51: SMBus disabled
pciide0 at pci0 dev 15 function 1 "ServerWorks OSB4 IDE" rev 0x00: DMA
atapiscsi0 at pciide0 channel 0 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <COMPAQ, CD-ROM SN-124, N102> SCSI0 5/cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 1
ohci0 at pci0 dev 15 function 2 "ServerWorks OSB4/CSB5 USB" rev 0x04: irq 11,
version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: ServerWorks OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
isa0 at mainbus0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ef4d netmask efed ttymask ffef
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
uhub0: device problem, disabling port 1
dkcsum: sd0 matches BIOS drive 0x80
dkcsum: sd1 matches BIOS drive 0x81
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
write-test.c:
* write-test.c
*
* write-test is meant to test simple writing performance to a *new*
* file. Blocksize can be set, default is 128KB.
*
* Usage:
*
* This program performs tests on a *new* file. (it will truncate an
* existing file.)
*
* Although performance results are returned when file-creation is finished,
* they don't have any relevance for random writes. File-creation is
* performed using an open(2) call.
*
* This program writes blocks of the specified block size, default=128KB.
*
* To perform a test:
*
* write-test -f <file-1>
*
* Compilation:
*
* cc -O4 -o write-test write-test.c
*
* Version: 1.0
*
* author: Robert Urban <[EMAIL PROTECTED]>
*-----------------------------------------------------------------------*/
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <signal.h>
#include <time.h>
#include <sys/time.h>
#include <errno.h>
#define KILOBYTE (1024)
#define MEGABYTE (1024 * 1024)
#define GIGABYTE (1024 * 1024 * 1024)
#define DEF_BLOCKSIZE (128 * 1024)
#define MAX_BLOCKSIZE (256 * 1024)
/* prototypes */
void handleSignal(int sig);
float diff(struct timeval *start, struct timeval *end);
void initChunk(char ch);
void usage();
void writeData();
/* globals */
int allrandom = 1;
int debug = 0;
long size = 128; /* 128 MB */
char filename[128] = { NULL };
int seconds = 60;
int io_count = 0;
float latency_cum = 0.0;
int blocksize = DEF_BLOCKSIZE;
long written = 0;
struct timeval bm_start, bm_end;
struct timeval io_start, io_end;
char *chunk;
int thefd;
int blockcount;
int f_sync = 0;
int f_directio = 0;
main(int argc, char *argv[]) {
time_t tloc;
char ran_args[32];
int c, errflg = 0;
struct stat statbuf;
int num_files = 0;
struct timeval tv;
gettimeofday(&tv, NULL);
srand48(tv.tv_usec);
#define ARGS "hdDSf:s:b:"
optarg = NULL;
while (!errflg && ((c = getopt(argc, argv, ARGS)) != -1))
switch (c) {
case 'h':
usage();
exit(0);
case 'b':
blocksize = atoi(optarg) * KILOBYTE;
break;
case 's':
/* size = atoi(optarg) * MEGABYTE; */
size = atoi(optarg);
break;
case 'd':
debug = 1;
break;
case 'S':
f_sync = 1;
break;
case 'f':
if (num_files == 1) {
printf("you cannot specify more than 1
file.\n");
exit(1);
}
strcpy(filename, optarg);
num_files++;
break;
default:
printf("unrecognized option: [%c].\n", c);
errflg++;
usage();
exit(-1);
}
chunk = (char *) malloc(MAX_BLOCKSIZE);
if (num_files != 1) {
printf("exactly 1 file must be specified using '-f'.\n");
exit(1);
}
/* convert size to bytes */
size *= MEGABYTE;
initChunk('A');
blockcount = size / blocksize;
if (debug) {
printf("SIZE : %ld\n", size);
printf("BLOCKS : %d\n", blockcount);
printf("BLKSIZ : %d\n", blocksize);
printf("FILE : %s\n", filename);
printf("SYNC : %s\n", f_sync ? "Yes" : "No");
}
if (filename == 0) {
printf("must supply a file using \"-f <filename>\"\n");
exit(-1);
}
signal(SIGINT, handleSignal);
signal(SIGTERM, handleSignal);
signal(SIGHUP, handleSignal);
signal(SIGALRM, handleSignal);
gettimeofday(&bm_start, NULL);
writeData();
handleSignal(0);
}
void usage() {
printf("usage: write-test -f <file> [flags]\n");
printf("\twhere flags are:\n");
printf("\t-d turn on debug messages\n");
printf("\t-S open output file with O_SYNC\n");
printf("\t-s <size> specify size in Megebytes
(default=100MB)\n");
printf("\t-b <blocksize> specify blocksize in Kilobytes
(default=128)\n");
}
void markStart()
{
gettimeofday(&io_start, NULL);
}
void markEnd()
{
float et;
gettimeofday(&io_end, NULL);
et = diff(&io_start, &io_end);
latency_cum += et;
if (debug) printf("latency: %.3f ms\n", et*1000);
}
void handleSignal(int sig) {
float et;
int index;
gettimeofday(&bm_end, NULL);
et = diff(&bm_start, &bm_end);
printf("%d IOs in %.2f seconds, %.2f IOs/sec, %.2f MB/sec\n",
io_count, et,
io_count/et,
((float)written/et)/(float)MEGABYTE);
printf("latency for %s: %.3f ms\n",
filename,
latency_cum/io_count*1000);
close(thefd);
exit(0);
}
void writeData() {
int this_blocksize;
int ret;
/* int flags = O_CREAT|O_TRUNC|O_WRONLY|O_SYNC; */
int flags = O_CREAT|O_TRUNC|O_WRONLY;
mode_t mode = 0666;
if (f_sync) {
flags |= O_SYNC;
}
if ((thefd = open(filename, flags, mode)) == -1) {
printf("open of \"%s\" failed for writing, errno=%d\n",
filename, errno);
exit(-1);
}
if (debug) printf("entering loop.\n");
while(written < size) {
this_blocksize = ((size - written) < blocksize) ?
(size - written) : blocksize;
markStart();
if (debug) printf("writing [%d] bytes...\n", this_blocksize);
ret = write(thefd, chunk, this_blocksize);
markEnd();
if (ret == -1) {
printf("write failed.\n");
exit(1);
}
written += this_blocksize;
io_count++;
}
handleSignal(0);
}
int spew(char *tag)
{
struct timeval tv;
gettimeofday(&tv, NULL);
printf(" TS[%s]: %lu.%d\n", tag, tv.tv_sec, tv.tv_usec);
return(0);
}
void initChunk(char ch) {
memset(chunk, ch, MAX_BLOCKSIZE);
}
float diff(struct timeval *start, struct timeval *end)
{
float d = end->tv_sec - start->tv_sec +
(end->tv_usec - start->tv_usec)/1000000.0;
return(d);
}
bioctl -iv ami0:
----------------------------------------------------------------
# bioctl -iv ami0
Volume Status Size Device
ami0 0 Online 218219151360 sd0 RAID5
0 Online 72834088960 0:0.0 noencl <COMPAQ BD07288277 HPB2>
'3KT06BVL000075237CP0'
1 Online 72834088960 0:1.0 noencl <COMPAQ BD07285A25 HPB8>
'3HZ6Z63500007433D7TT'
2 Online 72834088960 0:2.0 noencl <COMPAQ BD07287B4C HPB8>
'D20LF9XK'
3 Online 72834088960 0:3.0 noencl <COMPAQ BD07288277 HPB2>
'3KT06BXP00007523V3F7'
ami0 1 Online 299976622080 sd1 RAID0
0 Online 299995496448 0:5.0 noencl <COMPAQ BD30089BBA HPB1>
'DA01P5B007CG0547'
ami0 2 Hot spare 72834088960 0:4.0 noencl <COMPAQ BD0728A4B4 HPBC>
'J20JWRZK'
