On 22/08/2006, at 10:17 AM, Robert Urban wrote:

Hi Folks,

using a simple test program to write sequential blocks to a file, optionally opening with O_SYNC, I've tested write performance to a MegaRAID logical drive consisting of a RAID-5 set of 4 72GB HP Ultra320 disks and to a RAID-0
drive consisting of a single 300GB HP Ultra320 disk.  The controller
has 128MB of cache, but I do not have a battery, so cache write policy
is write-through. The kernel is the "bsd", and not "bsd.mp", but it makes
no difference. I've tried both.

bsd should be slightly faster than bsd.mp, but not enough for you to notice. Can you bring the box up to a snapshot?

All tests performed with O_SYNC, to avoid bufcache interaction.

if you really want to avoid that then use /dev/rsdXX.

The performance, at least to my perhaps naive eyes, seems abysmal. I'm getting
1.2MB/sec on the RAID-5 logical drive.

It doesn't matter what blocksize I choose, 32k, 64k, 128k, 256k. The chunksize
on the logical drive is the default, 64k.

On the single HP 300GB Ultra320 RAID-0 drive write performance is also bad:
1.7MB/sec.

The systems is (in case you hadn't already guessed) a Compaq ProLiant DL380-G2 with 1GB main memory and dual 1266MHz CPUs. I will attach dmesg, my test
program, and the output of "bioctl -iv ami0" below.

I can't imagine the MegaRAID controller is this slow. Any suggestions?

can you try running multiple copies of your test app at the same time and see if performance increases?

regards,
dlg


Rob Urban

dmesg:
---------------------------------------------------------------
OpenBSD 3.9 (GENERIC) #617: Thu Mar  2 02:26:48 MST 2006
    [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) III CPU family 1266MHz ("GenuineIntel" 686-class)
1.27 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE3 6,M
MX,FXSR,SSE
real mem  = 1073307648 (1048152K)
avail mem = 972660736 (949864K)
using 4278 buffers containing 53768192 bytes (52508K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf0000
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x2000
pcibios0: PCI BIOS has 9 Interrupt Routing table entries
pcibios0: PCI Interrupt Router at 000:15:0 ("ServerWorks OSB4" rev 0x00)
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x2600 0xee000/0x2000!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20HE Host" rev 0x23
pci1 at pchb0 bus 1
ppb0 at pci1 dev 3 function 0 "Intel i960 RP PCI-PCI" rev 0x05
pci2 at ppb0 bus 2
vga1 at pci2 dev 0 function 0 "ATI Mach64 GV" rev 0x7a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Intel 80960RP ATU" rev 0x05 at pci1 dev 3 function 1 not configured
pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20HE Host" rev 0x01
pchb2 at pci0 dev 0 function 2 "ServerWorks CNB20HE Host" rev 0x01
pchb3 at pci0 dev 0 function 3 "ServerWorks CNB20HE Host" rev 0x01
pci3 at pchb3 bus 7
ami0 at pci3 dev 4 function 0 "Symbios Logic MegaRAID" rev 0x01: irq 3 Dell
518 64b/lhc
ami0: FW 351X, BIOS v1.10, 128MB RAM
ami0: 2 channels, 0 FC loops, 2 logical drives
scsibus0 at ami0: 40 targets
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/ direct fixed sd0: 208110MB, 208110 cyl, 64 head, 32 sec, 512 bytes/sec, 426209280 sec total sd1 at scsibus0 targ 1 lun 0: <AMI, Host drive #01, > SCSI2 0/ direct fixed sd1: 286080MB, 286080 cyl, 64 head, 32 sec, 512 bytes/sec, 585891840 sec total
scsibus1 at ami0: 16 targets
uk0 at scsibus1 targ 15 lun 0: <COMPAQ, PROLIANT 4L6I, 1.84> SCSI2 3/processor
fixed
uk0: unknown device
scsibus2 at ami0: 16 targets
"Compaq PCI Hotplug" rev 0x12 at pci3 dev 7 function 0 not configured
fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x08, i82559: irq 5, address
00:08:02:8a:4b:fc
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
fxp1 at pci0 dev 4 function 0 "Intel 8255x" rev 0x08, i82559: irq 7, address
00:08:02:8a:4b:fb
inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4
"Compaq Netelligent ASMC" rev 0x00 at pci0 dev 6 function 0 not configured piixpm0 at pci0 dev 15 function 0 "ServerWorks OSB4" rev 0x51: SMBus disabled
pciide0 at pci0 dev 15 function 1 "ServerWorks OSB4 IDE" rev 0x00: DMA
atapiscsi0 at pciide0 channel 0 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <COMPAQ, CD-ROM SN-124, N102> SCSI0 5/ cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 1
ohci0 at pci0 dev 15 function 2 "ServerWorks OSB4/CSB5 USB" rev 0x04: irq 11,
version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: ServerWorks OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
isa0 at mainbus0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ef4d netmask efed ttymask ffef
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
uhub0: device problem, disabling port 1
dkcsum: sd0 matches BIOS drive 0x80
dkcsum: sd1 matches BIOS drive 0x81
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02

write-test.c:
 * write-test.c
 *
 * write-test is meant to test simple writing performance to a *new*
 * file.  Blocksize can be set, default is 128KB.
 *
 * Usage:
 *
 * This program performs tests on a *new* file. (it will truncate an
 * existing file.)
 *
* Although performance results are returned when file-creation is finished,
 * they don't have any relevance for random writes.  File-creation is
 * performed using an open(2) call.
 *
* This program writes blocks of the specified block size, default=128KB.
 *
 * To perform a test:
 *
 * write-test -f <file-1>
 *
 * Compilation:
 *
 * cc -O4 -o write-test write-test.c
 *
 * Version: 1.0
 *
 * author: Robert Urban <[EMAIL PROTECTED]>
*--------------------------------------------------------------------- --*/

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <signal.h>
#include <time.h>
#include <sys/time.h>
#include <errno.h>

#define KILOBYTE                        (1024)
#define MEGABYTE                        (1024 * 1024)
#define GIGABYTE                        (1024 * 1024 * 1024)
#define DEF_BLOCKSIZE           (128 * 1024)
#define MAX_BLOCKSIZE           (256 * 1024)


/* prototypes */
void handleSignal(int sig);
float diff(struct timeval *start, struct timeval *end);
void initChunk(char ch);
void usage();
void writeData();


/* globals */
int allrandom                           = 1;
int debug                                       = 0;
long size                                       = 128;          /* 128 MB */
char filename[128]                      = { NULL };
int seconds                                     = 60;
int io_count                            = 0;
float latency_cum                       = 0.0;
int blocksize                           = DEF_BLOCKSIZE;
long written                            = 0;

struct timeval bm_start, bm_end;
struct timeval io_start, io_end;

char *chunk;
int thefd;
int blockcount;
int f_sync = 0;
int f_directio = 0;

main(int argc, char *argv[]) {
        time_t tloc;
        char ran_args[32];
        int c, errflg = 0;
        struct stat statbuf;
        int num_files = 0;
        struct timeval tv;

        gettimeofday(&tv, NULL);
        srand48(tv.tv_usec);

#define ARGS "hdDSf:s:b:"

        optarg = NULL;
        while (!errflg && ((c = getopt(argc, argv, ARGS)) != -1))
                switch (c) {
                        case 'h':
                                usage();
                                exit(0);
                        case 'b':
                                blocksize = atoi(optarg) * KILOBYTE;
                                break;
                        case 's':
                                /* size = atoi(optarg) * MEGABYTE; */
                                size = atoi(optarg);
                                break;
                        case 'd':
                                debug = 1;
                                break;
                        case 'S':
                                f_sync = 1;
                                break;
                        case 'f':
                                if (num_files == 1) {
                                        printf("you cannot specify more than 1 
file.\n");
                                        exit(1);
                                }
                                strcpy(filename, optarg);
                                num_files++;
                                break;
                        default:
                                printf("unrecognized option: [%c].\n", c);
                                errflg++;
                                usage();
                                exit(-1);
                }

        chunk = (char *) malloc(MAX_BLOCKSIZE);

        if (num_files != 1) {
                printf("exactly 1 file must be specified using '-f'.\n");
                exit(1);
        }

        /* convert size to bytes */
        size *= MEGABYTE;

        initChunk('A');

        blockcount = size / blocksize;

        if (debug) {
                printf("SIZE      : %ld\n", size);
                printf("BLOCKS    : %d\n", blockcount);
                printf("BLKSIZ    : %d\n", blocksize);
                printf("FILE      : %s\n", filename);
                printf("SYNC      : %s\n", f_sync ? "Yes" : "No");
        }

        if (filename == 0) {
                printf("must supply a file using \"-f <filename>\"\n");
                exit(-1);
        }

        signal(SIGINT, handleSignal);
        signal(SIGTERM, handleSignal);
        signal(SIGHUP, handleSignal);
        signal(SIGALRM, handleSignal);

        gettimeofday(&bm_start, NULL);
        writeData();
        handleSignal(0);
}

void usage() {
        printf("usage: write-test -f <file> [flags]\n");
        printf("\twhere flags are:\n");
        printf("\t-d                turn on debug messages\n");
        printf("\t-S                open output file with O_SYNC\n");
printf("\t-s <size> specify size in Megebytes (default=100MB)\n"); printf("\t-b <blocksize> specify blocksize in Kilobytes (default=128)\n");
}

void markStart()
{
        gettimeofday(&io_start, NULL);
}

void markEnd()
{
        float et;
        gettimeofday(&io_end, NULL);

        et = diff(&io_start, &io_end);
        latency_cum += et;
        if (debug) printf("latency: %.3f ms\n", et*1000);
}

void handleSignal(int sig) {
        float et;
        int index;

        gettimeofday(&bm_end, NULL);
        et = diff(&bm_start, &bm_end);
        printf("%d IOs in %.2f seconds, %.2f IOs/sec, %.2f MB/sec\n",
                io_count, et,
                io_count/et,
                ((float)written/et)/(float)MEGABYTE);

        printf("latency for %s: %.3f ms\n",
                filename,
                latency_cum/io_count*1000);

        close(thefd);
        exit(0);
}

void writeData() {
        int this_blocksize;
        int ret;
        /* int flags = O_CREAT|O_TRUNC|O_WRONLY|O_SYNC; */
        int flags = O_CREAT|O_TRUNC|O_WRONLY;
        mode_t mode = 0666;

        if (f_sync) {
                flags |= O_SYNC;
        }

        if ((thefd = open(filename, flags, mode)) == -1) {
                printf("open of \"%s\" failed for writing, errno=%d\n",
                        filename, errno);
                exit(-1);
        }

        if (debug) printf("entering loop.\n");

        while(written < size) {

                this_blocksize = ((size - written) < blocksize) ?
                        (size - written) : blocksize;

                markStart();
                if (debug) printf("writing [%d] bytes...\n", this_blocksize);
                ret = write(thefd, chunk, this_blocksize);
                markEnd();
                if (ret == -1) {
                        printf("write failed.\n");
                        exit(1);
                }
                written += this_blocksize;
                io_count++;
        }

        handleSignal(0);
}

int spew(char *tag)
{
        struct timeval tv;

        gettimeofday(&tv, NULL);
        printf("  TS[%s]: %lu.%d\n", tag, tv.tv_sec, tv.tv_usec);

        return(0);
}

void initChunk(char ch) {
        memset(chunk, ch, MAX_BLOCKSIZE);
}

float diff(struct timeval *start, struct timeval *end)
{
        float d = end->tv_sec - start->tv_sec +
                (end->tv_usec - start->tv_usec)/1000000.0;

        return(d);
}

bioctl -iv ami0:
----------------------------------------------------------------
# bioctl -iv ami0
Volume  Status     Size           Device
 ami0 0 Online       218219151360 sd0     RAID5
0 Online 72834088960 0:0.0 noencl <COMPAQ BD07288277 HPB2> '3KT06BVL000075237CP0' 1 Online 72834088960 0:1.0 noencl <COMPAQ BD07285A25 HPB8> '3HZ6Z63500007433D7TT' 2 Online 72834088960 0:2.0 noencl <COMPAQ BD07287B4C HPB8>
                                                 'D20LF9XK'
3 Online 72834088960 0:3.0 noencl <COMPAQ BD07288277 HPB2> '3KT06BXP00007523V3F7'
 ami0 1 Online       299976622080 sd1     RAID0
0 Online 299995496448 0:5.0 noencl <COMPAQ BD30089BBA HPB1>
                                                 'DA01P5B007CG0547'
ami0 2 Hot spare 72834088960 0:4.0 noencl <COMPAQ BD0728A4B4 HPBC>
                                                 'J20JWRZK'

Reply via email to