Petri -- thanks for the idea.
I ran 2 dds in parallel; they took roughly twice as long in clock
time, and had about 1/2 the throughput of the single dd. On my
system it doesn't look like how the work is offered to the disk
subsystem matters.
# time dd if=/dev/zero of=blort1 bs=1m count=1000 & ; time dd
if=/dev/zero of=blort2 bs=1m count=1000&
[1] 69356
[2] 69357
#
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 382.511089 secs (2741296 bytes/sec)
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 385.677897 secs (2718787 bytes/sec)
[2] Done dd if=/dev/zero of=blort2 bs=1m count=1000
0.023u 16.913s 6:25.75 4.3% 20+2813k 3+8772io 2pf+0w
[1] + Done dd if=/dev/zero of=blort1 bs=1m count=1000
0.023u 16.880s 6:22.70 4.4% 20+2810k 1+8765io 0pf+0w
vmstat output during this run:
procs memory page disks
faults cpu
r b w avm fre flt re pi po fr sr ad0 ad4 ad6 ad8
ad10 in sy cs us sy id
1 1 0 101692 13132 0 0 0 0 5872 1857 0 54 62 69 59
2293 321 3914 2 62 35
3 1 0 101692 15576 0 4 0 0 5840 1947 0 52 54 60 57
2023 308 3379 2 64 35
2 1 0 101692 10072 0 0 0 0 5344 0 0 61 54 55 56
2133 368 3496 0 58 42
2 2 0 101692 11496 0 0 0 0 5984 1860 0 61 65 57 61
2377 429 4118 1 69 30
3 0 0 101692 13124 0 0 0 0 5568 1799 0 57 62 62 65
2440 466 4252 1 64 36
1 2 0 101692 12404 0 0 0 0 6176 1868 0 61 70 67 59
1923 482 3337 1 64 36
0 3 0 101692 15080 0 0 0 0 6576 1805 0 67 71 65 68
2423 468 4300 1 68 31
0 2 0 101692 17448 0 0 0 0 5760 1792 0 56 57 57 60
2463 399 4155 4 65 31
That got me thinking about whether there was a bottleneck to the disk
somewhere else. I ran a dd reading 1GB simultaneous from each of the
4 disks. Each disk could be read at about 25MB/sec for a combined
read transfer rate of 100MB/sec. A snipped of iostat output:
tty ad0 ad4 ad6
ad8 ad10 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t
tps MB/s KB/t tps MB/s us ni sy in id
0 334 0.00 0 0.00 127.36 201 24.97 126.73 201
24.85 127.35 198 24.60 126.71 199 24.60 2 0 11 12 75
0 115 0.00 0 0.00 128.00 191 23.86 126.98 188
23.30 128.00 196 24.48 127.36 199 24.73 2 0 15 9 73
0 115 0.00 0 0.00 127.37 203 25.23 127.08 208
25.79 126.75 205 25.35 127.27 175 21.73 2 0 9 9 80
So I'm definitely not getting the full 60MB/sec performance out of
the disk with my current controller and mobo.
I guess the fundamental question is this -- if I have a 4 disk
subsystem that supports an aggregate ~100MB/sec transfer raw to the
underlying disks, is it reasonable to expect a ~5MB/sec transfer rate
for a RAID5 hosted on that subsystem -- a 95% overhead.
Steve
At 01:19 PM 10/28/2006, Petri Helenius wrote:
According to my understanding vinum does not overlap requests to
multiple disks when running in raid5 configuration so you're not
going to achieve good numbers with just "single stream" tests.
Pete
Steve Peterson wrote:
Eric -- thanks for looking at my issue. Here's a dd reading from
one of the disks underlying the array (the others have basically
the same performance):
$ time dd if=/dev/ad10 of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 15.322421 secs (68434095 bytes/sec)
0.008u 0.506s 0:15.33 3.2% 20+2715k 0+0io 0pf+0w
and here's a dd reading from the raw gvinum device /dev/gvinum/vol1:
$ time dd if=/dev/gvinum/vol1 of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 25.870684 secs (40531437 bytes/sec)
0.006u 0.552s 0:25.88 2.1% 23+3145k 0+0io 0pf+0w
Is there a way to nondestructively write to the raw disk or gvinum device?
For comparison, here's a read against the raw PATA device on the machine:
$ time dd if=/dev/ad0 of=/dev/null bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 26.096070 secs (40181376 bytes/sec)
0.013u 0.538s 0:26.10 2.0% 24+3322k 0+0io 0pf+0w
Steve
At 11:14 PM 10/27/2006, Eric Anderson wrote:
On 10/27/06 18:03, Steve Peterson wrote:
I recently set up a media server for home use and decided to try
the gvinum raid5 support to learn about it and see how it
performs. It seems slower than I'd expect -- a little under
6MB/second, with about 50 IOs/drive/second -- and I'm trying to
understand why. Any assistance/pointers would be appreciated.
The disk system consists of 4 Seagate NL35 SATA ST3250623NS
drives connected to a Promise TX4300 (PDC40719) controller,
organized as a RAID5 volume via gvinum using this configuration:
drive drive01 device /dev/ad10
drive drive02 device /dev/ad6
drive drive03 device /dev/ad4
drive drive04 device /dev/ad8
volume vol1
plex org raid5 256k
sd length 200001m drive drive01
sd length 200001m drive drive02
sd length 200001m drive drive03
sd length 200001m drive drive04
dd reports the following performance on a 1G file write to the
RAID5 hosted volume:
$ time dd if=/dev/zero of=big.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec)
179.76 real 0.02 user 16.60 sys
By comparison, creating the same file on the system disk (an old
ATA ST380021A connected via a SIS 730 on the motherboard):
time dd if=/dev/zero of=big.file bs=1m count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec)
28.32 real 0.01 user 19.13 sys
and vmstat reports about 280-300 IOs/second to that drive.
The CPU is pretty weak -- an Athlon 750. Is that the source of
my problem? If you look at the vmstat output below the machine
is busy but not pegged.
Try the dd to the raw gvinum device instead of through a
filesystem, and also to the individual disks. That will at least
tell us where to look.
Eric
--
------------------------------------------------------------------------
Eric Anderson Sr. Systems Administrator Centaur Technology
Anything that works is better than anything that doesn't.
------------------------------------------------------------------------
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to
"[EMAIL PROTECTED]"
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to
"[EMAIL PROTECTED]"
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
