Re: (S)ATA performance in FBSD 6.2/7.0
On Fri, 2 Mar 2007, O. Hartmann wrote: On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to another while the box didn't do anything else than copying. I watched the copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' never go byond 128 (128kb buffer limit?). But more frustrating, I never got beyond 33 MB/s transfer rate although bonni/bonni++ told me both drives are capable doing much more (~75 MB/s each). At home, I use a FreeBSD 7.0-CURRENT box on an ASUS A8N32-SLI/nForce4-SLI based box, amd64 (no 32Bit compatibility). Two Hitachi T7K250 250 GB/SATA II drives build up a RAID 0 (nVidia MediaShield), and additionally there is a SAMSUNG Spinpoitn SP2004C attached to the controller. bonni results in 55 MB/s for the SP2004C alone and gives ~ 65 - 70 MB/s for the Hitachis, each and roughly 115 MB/s for the RAID 0. But copying from the single drive to the RAID 0 or from the RAID 0 to the single drive also reaches this oscure 33 MB/s boundary! It would be helpful if you could provide the dmesg output for the ATA controllers and devices. Sometimes, issues such as this come down to negotiated speed and bus topology. For example, if you have two devices on the same bus, then each byte of data copied has to pass over the bus twice, which may run into performance limits on the bus where moving the data over the bus once won't. Another area to investigate is the difference between raw device performance and file system performance. Does it make a difference if you copy using dd to a file that is already allocated? What performance does dd get when just reading from the file, or using dd to copy the file? What if the file already exists and you are simply overwriting it (avoiding the overhead of allocating)? Is your disk fragmented, leading to more seeks in placing new files, or is it relatively empty, which should lead to more contiguous storage? Finally, disk performance varies enourmously by location where the I/O is taking place on the disk. Try running diskinfo -t on your various disk devices, and see if there's a correlation between the maximum I/O rate you're experiencing and the location on the source or target drive? On the ATA disk on my workstation, I see 53M/s on the outside of the disk platter, but only 31M/s on the inside of the disk platter. Especially if your drive is relatively full, files may be ending up in less preferred locations on the disk resulting in reduced performance. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
Mike Tancsa wrote: At 04:38 AM 3/2/2007, O. Hartmann wrote: The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to Something strange about your setup I would say. I just tried on a Segate SATA drive off an ICH5 chipset (plain old P IV 2.4Ghz). Do you have an option in your BIOS for native mode or compatibility mode for the SATA controller ? If so, try toggling that to native SATA mode [ns4]% iostat -c 1000 tty ad4twed0 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 2 447 4.91 0 0.00 23.77 40 0.92 20 0 6 0 74 4 307 0.00 0 0.00 12.61 14 0.17 0 0 0 0 100 1 183 0.00 0 0.00 14.50 4 0.06 0 0 0 0 100 1 63 128.00 47 5.82 0.00 0 0.00 7 0 7 0 86 0 182 128.00 534 66.70 15.25 8 0.12 0 0 15 8 77 0 60 128.00 553 69.13 2.00 2 0.00 0 0 8 8 85 0 182 128.00 537 67.14 14.50 4 0.06 15 0 31 15 38 0 60 128.00 553 69.06 0.00 0 0.00 54 0 0 8 38 0 60 128.00 538 67.21 0.00 0 0.00 23 0 0 8 69 1 301 128.00 495 61.88 12.18 22 0.26 0 0 8 0 92 [ns4]# dd if=/dev/ad4 of=/dev/null bs=1024k ^C410+0 records in 410+0 records out 429916160 bytes transferred in 6.089321 secs (70601659 bytes/sec) [ns4]# [ns4]# atacontrol cap ad4 Protocol Serial ATA II device model ST3400833NS serial number 5NF25DTG firmware revision 3.AEH cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 781422768 sectors dma supported overlap not supported Feature Support EnableValue Vendor write cacheyes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management no no 65278/0xFEFE automatic acoustic management no no 0/0x00 254/0xFE [ns4]# Sorry for waiting so long, but weekend and a lot of work prevented me from answering earlier. All right, I checked on three boxes the BIOS settings and ensured no legacy settings on any SATA device. In general, I realized using 'dd' results in much better drive-to-drive bandwith than doing just a 'cp'. As many times recommended herein, I used bs=128k, but look for yourself on the results. Box A, FreeBSD 7.0-CURRENT/AMD64: FreeBSD foo.org 7.0-CURRENT FreeBSD 7.0-CURRENT #38: Sun Mar 4 12:45:42 CET 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/FOO amd64 Relevant harware: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-CURRENT #38: Sun Mar 4 12:45:42 CET 2007 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/THOR ACPI APIC Table: A M I OEMAPIC Timecounter i8254 frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3500+ (2210.07-MHz K8-class CPU) Origin = AuthenticAMD Id = 0x10ff0 Stepping = 0 Features=0x78bfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2 AMD Features=0xe2500800SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow! AMD Features2=0x1LAHF usable memory = 2136956928 (2037 MB) avail memory = 2062622720 (1967 MB) ioapic0 Version 1.1 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: A M I OEMXSDT on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter ACPI-safe frequency 3579545 Hz quality 1000 acpi_timer0: 24-bit timer at 3.579545MHz port 0x508-0x50b on acpi0 cpu0: ACPI CPU on acpi0 [...] atapci0: nVidia nForce CK804 UDMA133 controller port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 15.0 on pci0 ata0: ATA channel 0 on atapci0 ata0: [ITHREAD] ata1: ATA channel 1 on atapci0 ata1: [ITHREAD] atapci1: nVidia nForce CK804 SATA300 controller port 0xb480-0xb487,0xb400-0xb403,0xb080-0xb087,0xb000-0xb003,0xac00-0xac0f mem 0xddcf9000-0xddcf9fff irq 23 at device 16.0 on pci0 atapci1: [ITHREAD] ata2: ATA channel 0 on atapci1 ata2: [ITHREAD] ata3: ATA channel 1 on atapci1 ata3: [ITHREAD] atapci2: nVidia nForce CK804 SATA300 controller port 0xa880-0xa887,0xa800-0xa803,0xa480-0xa487,0xa400-0xa403,0xa080-0xa08f mem 0xddcf8000-0xddcf8fff irq 20 at device 17.0 on pci0 atapci2: [ITHREAD] ata4: ATA
Re: (S)ATA performance in FBSD 6.2/7.0
- Original Message - From: Cheffo [EMAIL PROTECTED] To: Ted Mittelstaedt [EMAIL PROTECTED] Cc: freebsd-performance@freebsd.org; O. Hartmann [EMAIL PROTECTED]; freebsd-questions@freebsd.org Sent: Friday, March 02, 2007 3:38 AM Subject: Re: (S)ATA performance in FBSD 6.2/7.0 Hi, I think the problem is that the benchmark runs with small files and most files are in cache that's why it shows higher speeds I think the problem is O Hartmann (the OP) has proven by his lack of followup that he was just trolling. Kind of what I guessed yesterday. Ted ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
(S)ATA performance in FBSD 6.2/7.0
The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to another while the box didn't do anything else than copying. I watched the copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' never go byond 128 (128kb buffer limit?). But more frustrating, I never got beyond 33 MB/s transfer rate although bonni/bonni++ told me both drives are capable doing much more (~75 MB/s each). At home, I use a FreeBSD 7.0-CURRENT box on an ASUS A8N32-SLI/nForce4-SLI based box, amd64 (no 32Bit compatibility). Two Hitachi T7K250 250 GB/SATA II drives build up a RAID 0 (nVidia MediaShield), and additionally there is a SAMSUNG Spinpoitn SP2004C attached to the controller. bonni results in 55 MB/s for the SP2004C alone and gives ~ 65 - 70 MB/s for the Hitachis, each and roughly 115 MB/s for the RAID 0. But copying from the single drive to the RAID 0 or from the RAID 0 to the single drive also reaches this oscure 33 MB/s boundary! In the first place I thought the older i386 hardware has some hard-limits, but we have several boxes of the exact same hardware around here and a wide spread Linux and Windows utilization and on those boxes equipted with more than one harddrive (PATA or SATA) the effective transfer rate shown up is about 50 - 65 MB/s as expected with copying a big 5G file from one drive to another. The hardwrae limit is completely nonsense when it comes to the AMD64 box with newer hardware. Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 defaults, but on both boxes nForce4 and ICH5 controller are recognized and show up with SATA300 or SATA150 capabilities, respective)? May I have some knobs I'm not aware of to tune disk performance? I would appreciate any coments on that and if someone has some good ideas how to benchmark those subjects, please let me know. Regards, Oliver -- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
another while the box didn't do anything else than copying. I watched the copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' never go byond 128 (128kb buffer limit?). But more frustrating, I never got what's wrong? FreeBSD uses 128k limit by default. edit /usr/src/sys/sys/param.h and change #define MAXPHYS (128 * 1024) /* max raw I/O transfer size */ to say #define MAXPHYS (1024 * 1024) /* max raw I/O transfer size */ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
- Original Message - From: O. Hartmann [EMAIL PROTECTED] To: freebsd-performance@freebsd.org; freebsd-questions@freebsd.org Sent: Friday, March 02, 2007 1:38 AM Subject: (S)ATA performance in FBSD 6.2/7.0 The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. blah blah blah deleted Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 man mount read section on async linux by default mounts async freebsd by default mounts sync you can change FBSD to async then watch your fs scramble during a power failure no big deal, it's only your data. Ted ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
you can change FBSD to async then watch your fs scramble during a power failure no big deal, it's only your data. you are wrong, he talked about copying BIG files, and this shouldn't make a difference contrary to small files. there is something wrong there as i routinely get 70MB/s on my SATA server ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
--- O. Hartmann [EMAIL PROTECTED] wrote: Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 defaults, but on both boxes nForce4 and ICH5 controller are recognized and show up with SATA300 or SATA150 capabilities, respective)? May I have some knobs I'm not aware of to tune disk performance? I think, this 33MB/sec limit comes like this: The regular copy process (I think u used cp) reads with speed S from disk A and writes with speed S to disk B. But: While it reads, it doesnt write AND while it writes, it doesnt read. So you might want to try this: dd if=/diskA/fileA bs=128k | dd of=/diskB/fileB bs=128k You could also try just to read or to write. Or to readwrite with _independent_ processes. -Arne The fish are biting. Get more visitors on your site using Yahoo! Search Marketing. http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
Quoting Cheffo [EMAIL PROTECTED] (from Fri, 02 Mar 2007 13:38:45 +0200): Hi, Ted Mittelstaedt wrote: - Original Message - From: O. Hartmann [EMAIL PROTECTED] To: freebsd-performance@freebsd.org; freebsd-questions@freebsd.org Sent: Friday, March 02, 2007 1:38 AM Subject: (S)ATA performance in FBSD 6.2/7.0 The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. blah blah blah deleted Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 man mount read section on async linux by default mounts async freebsd by default mounts sync you can change FBSD to async then watch your fs scramble during a power failure no big deal, it's only your data. Ted If SYNC is default how can you explain this: [12:[EMAIL PROTECTED]:~# mount /dev/ad4s3a on / (ufs, local, synchronous) devfs on /dev (devfs, local) /dev/ad4s3d on /tmp (ufs, local, soft-updates) /dev/ad4s3f on /usr (ufs, local, soft-updates) /dev/ad4s3e on /var (ufs, local, soft-updates) [...] So I'm pretty sure that for type ufs async is default. Both of you are wrong. By default noasync is used. This is different from sync and async. Feel free to look up the difference. Also I do not see why sync should report different speeds for copy and benchmark tools if they do the same thing? Because cp may behave differently than the tools used to benchmark. A dd may be more portable in this case. Just to be sure I added to my /tmp entry async in /etc/fstab: /dev/ad4s3d /tmpufs rw,async 2 2 umounted and mounted again and still have: /dev/ad4s3d on /tmp (ufs, local, soft-updates) IIRC when SU is used, async is not used even if specified. But I' not sure about this. Asides from the linux async-by-default there's maybe also the write-cache-off penalty in FreeBSD. But I'm not sure it is off by default. I disable the WC myself in loader.conf everywhere to be on the safe side and I don't feel like experimenting ATM (I'm ill in bed). If the same conditions are tested in FreeBSD and linux (which is not easy, as we don't share a common FS implementation, even when we support the same FS type) and the sync/async and WC related stuff can be ruled out, it may be a problem in the (S)ATA code and it would be nice if we would know about this. So please dig deeper into this (it can also be a problem with our cp or GEOM or whatever). Bye, Alexander. -- I heard one time you single-handedly defeated a hoard of rampaging of somethings in the something something system. -Fry http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
Hi,
Ted Mittelstaedt wrote:
- Original Message -
From: O. Hartmann [EMAIL PROTECTED]
To: freebsd-performance@freebsd.org; freebsd-questions@freebsd.org
Sent: Friday, March 02, 2007 1:38 AM
Subject: (S)ATA performance in FBSD 6.2/7.0
The last days I tried to figure out why some of my lab's FreeBSD boxes
and also mine at home seem to be outperformed by some Linux setups
around here and I saw something interesting.
blah blah blah deleted
Before digging into this problem deeper with benchmarks, could anyone
explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33
man mount
read section on async
linux by default mounts async
freebsd by default mounts sync
you can change FBSD to async
then watch your fs scramble during a power failure
no big deal, it's only your data.
Ted
If SYNC is default how can you explain this:
[12:[EMAIL PROTECTED]:~# mount
/dev/ad4s3a on / (ufs, local, synchronous)
devfs on /dev (devfs, local)
/dev/ad4s3d on /tmp (ufs, local, soft-updates)
/dev/ad4s3f on /usr (ufs, local, soft-updates)
/dev/ad4s3e on /var (ufs, local, soft-updates)
[13:[EMAIL PROTECTED]:~# cat /etc/fstab
# DeviceMountpoint FStype Options Dump
Pass#
/dev/ad4s3b noneswapsw 0 0
/dev/ad4s3a / ufs rw,sync 1 1
/dev/ad4s3d /tmpufs rw 2 2
/dev/ad4s3f /usrufs rw 2 2
/dev/ad4s3e /varufs rw 2 2
And this is only because I manually add *sync* to my /etc/fstab.
E.g if sync is default why mount do not report that my /dev/ad4s3f on
/usr is mounted synchronous?
From what I seed in rc.X scripts mount -a -t ${mount_excludes} is used
to mount things form fstab at boot time (sync or async is not set
anywhere so we use dafault options here)
So I'm pretty sure that for type ufs async is default.
Also I do not see why sync should report different speeds for copy and
benchmark tools if they do the same thing?
Just to be sure I added to my /tmp entry async in /etc/fstab:
/dev/ad4s3d /tmpufs rw,async 2 2
umounted and mounted again and still have:
/dev/ad4s3d on /tmp (ufs, local, soft-updates)
I think the problem is that the benchmark runs with small files and most
files are in cache that's why it shows higher speeds - try to run
bonnie++ with more and bigger files to be sure that the cache is not
enough and to be able to see the real performance of your HDDs.
PS:
Here is what I got from RAID10 4x160GB SATA2 HDDs (areca RAID)
BLAH - bonnie++ -d /var/tmp -u root -s 16g -n 256:65536:65536:16
Version 1.93c --Sequential Output-- --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
blah.cmotd.com 16G 159 88 54264 24 24727 12 299 94 70744 19
223.5 12
Latency 63581us 803ms1123ms 93936us 94991us
251ms
Version 1.93c --Sequential Create-- Random
Create
blah.cmotd.com -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
256:65536:65536/16 715 24 826 25 17321 49 733 2451 2
6039 70
Latency 1220ms 408ms2805ms1189ms 692ms
2735ms
50MB/s write 70MB/s read from 4HDDs .. so I do not know how you expect
75MB/s with single HDD.
--
Best Wishes,
Stefan Lambrev
ICQ# 24134177
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
On Fri, Mar 02, 2007 at 10:43:34AM +0100, Wojciech Puchar wrote: another while the box didn't do anything else than copying. I watched the copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' never go byond 128 (128kb buffer limit?). But more frustrating, I never got what's wrong? FreeBSD uses 128k limit by default. edit /usr/src/sys/sys/param.h and change #define MAXPHYS (128 * 1024) /* max raw I/O transfer size */ to say #define MAXPHYS (1024 * 1024) /* max raw I/O transfer size */ did anyone measure impact on various benchmark of this change? is 128k the optimal size for nowadays computers ? if we can squeeze more performance out of a typical box by just raising one define it would be great... roman ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
On 03/02/07 06:03, Alexander Leidinger wrote: Quoting Cheffo [EMAIL PROTECTED] (from Fri, 02 Mar 2007 13:38:45 +0200): Hi, Ted Mittelstaedt wrote: - Original Message - From: O. Hartmann [EMAIL PROTECTED] To: freebsd-performance@freebsd.org; freebsd-questions@freebsd.org Sent: Friday, March 02, 2007 1:38 AM Subject: (S)ATA performance in FBSD 6.2/7.0 The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. blah blah blah deleted Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 man mount read section on async linux by default mounts async freebsd by default mounts sync you can change FBSD to async then watch your fs scramble during a power failure no big deal, it's only your data. Ted If SYNC is default how can you explain this: [12:[EMAIL PROTECTED]:~# mount /dev/ad4s3a on / (ufs, local, synchronous) devfs on /dev (devfs, local) /dev/ad4s3d on /tmp (ufs, local, soft-updates) /dev/ad4s3f on /usr (ufs, local, soft-updates) /dev/ad4s3e on /var (ufs, local, soft-updates) [...] So I'm pretty sure that for type ufs async is default. Both of you are wrong. By default noasync is used. This is different from sync and async. Feel free to look up the difference. Also I do not see why sync should report different speeds for copy and benchmark tools if they do the same thing? Because cp may behave differently than the tools used to benchmark. A dd may be more portable in this case. Just to be sure I added to my /tmp entry async in /etc/fstab: /dev/ad4s3d /tmpufs rw,async 2 2 umounted and mounted again and still have: /dev/ad4s3d on /tmp (ufs, local, soft-updates) IIRC when SU is used, async is not used even if specified. But I' not sure about this. Asides from the linux async-by-default there's maybe also the write-cache-off penalty in FreeBSD. But I'm not sure it is off by default. I disable the WC myself in loader.conf everywhere to be on the safe side and I don't feel like experimenting ATM (I'm ill in bed). If the same conditions are tested in FreeBSD and linux (which is not easy, as we don't share a common FS implementation, even when we support the same FS type) and the sync/async and WC related stuff can be ruled out, it may be a problem in the (S)ATA code and it would be nice if we would know about this. So please dig deeper into this (it can also be a problem with our cp or GEOM or whatever). People should not be using file system tools to measure hardware speeds like SATA or disks. That doesn't make sense, since a portion of that benchmark would then include the file system, which as you mention is very very different between OS'es. cp shouldn't be used. dd is ok for bare minimum testing I suppose. On one of my SATA memory disks, I can get 125MB/s through it, with no extra magic. Eric ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
On Fri, Mar 02, 2007 at 10:38:35AM +0100, O. Hartmann wrote: The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to another while the box didn't do anything else than copying. I watched the copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' never go byond 128 (128kb buffer limit?). But more frustrating, I never got beyond 33 MB/s transfer rate although bonni/bonni++ told me both drives are capable doing much more (~75 MB/s each). At home, I use a FreeBSD 7.0-CURRENT box on an ASUS A8N32-SLI/nForce4-SLI based box, amd64 (no 32Bit compatibility). Two Hitachi T7K250 250 GB/SATA II drives build up a RAID 0 (nVidia MediaShield), and additionally there is a SAMSUNG Spinpoitn SP2004C attached to the controller. bonni results in 55 MB/s for the SP2004C alone and gives ~ 65 - 70 MB/s for the Hitachis, each and roughly 115 MB/s for the RAID 0. But copying from the single drive to the RAID 0 or from the RAID 0 to the single drive also reaches this oscure 33 MB/s boundary! In the first place I thought the older i386 hardware has some hard-limits, but we have several boxes of the exact same hardware around here and a wide spread Linux and Windows utilization and on those boxes equipted with more than one harddrive (PATA or SATA) the effective transfer rate shown up is about 50 - 65 MB/s as expected with copying a big 5G file from one drive to another. The hardwrae limit is completely nonsense when it comes to the AMD64 box with newer hardware. Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 defaults, but on both boxes nForce4 and ICH5 controller are recognized and show up with SATA300 or SATA150 capabilities, respective)? May I have some knobs I'm not aware of to tune disk performance? I would appreciate any coments on that and if someone has some good ideas how to benchmark those subjects, please let me know. One thing to keep in mind is that it matters a lot were on the disk you place the data due to the higher angular density of data at the outside of the disk. The results you are seeing are close to consistant with the kind of results I'd expect to see from writing at opposiste edges of the disk. The 33MB/s is suspious ane may diserve investigation, but make sure you are writing to the same part of the disk if you want to compare disk IO rates. There's an example of IO rates on recent large SATA disks: http://storagereview.com/articles/200607/500_2.html Also, you should time the actual copy and do the math to verify that vmstat is actually producing valid results. It's possible there's a bug in vmstat or the underlying statistics it uses. -- Brooks pgp3kTgCZ7ZtA.pgp Description: PGP signature
Re: (S)ATA performance in FBSD 6.2/7.0
On 03/02/07 09:28, Brooks Davis wrote: On Fri, Mar 02, 2007 at 10:38:35AM +0100, O. Hartmann wrote: The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to another while the box didn't do anything else than copying. I watched the copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' never go byond 128 (128kb buffer limit?). But more frustrating, I never got beyond 33 MB/s transfer rate although bonni/bonni++ told me both drives are capable doing much more (~75 MB/s each). At home, I use a FreeBSD 7.0-CURRENT box on an ASUS A8N32-SLI/nForce4-SLI based box, amd64 (no 32Bit compatibility). Two Hitachi T7K250 250 GB/SATA II drives build up a RAID 0 (nVidia MediaShield), and additionally there is a SAMSUNG Spinpoitn SP2004C attached to the controller. bonni results in 55 MB/s for the SP2004C alone and gives ~ 65 - 70 MB/s for the Hitachis, each and roughly 115 MB/s for the RAID 0. But copying from the single drive to the RAID 0 or from the RAID 0 to the single drive also reaches this oscure 33 MB/s boundary! In the first place I thought the older i386 hardware has some hard-limits, but we have several boxes of the exact same hardware around here and a wide spread Linux and Windows utilization and on those boxes equipted with more than one harddrive (PATA or SATA) the effective transfer rate shown up is about 50 - 65 MB/s as expected with copying a big 5G file from one drive to another. The hardwrae limit is completely nonsense when it comes to the AMD64 box with newer hardware. Before digging into this problem deeper with benchmarks, could anyone explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 defaults, but on both boxes nForce4 and ICH5 controller are recognized and show up with SATA300 or SATA150 capabilities, respective)? May I have some knobs I'm not aware of to tune disk performance? I would appreciate any coments on that and if someone has some good ideas how to benchmark those subjects, please let me know. One thing to keep in mind is that it matters a lot were on the disk you place the data due to the higher angular density of data at the outside of the disk. The results you are seeing are close to consistant with the kind of results I'd expect to see from writing at opposiste edges of the disk. The 33MB/s is suspious ane may diserve investigation, but make sure you are writing to the same part of the disk if you want to compare disk IO rates. There's an example of IO rates on recent large SATA disks: http://storagereview.com/articles/200607/500_2.html Also, you should time the actual copy and do the math to verify that vmstat is actually producing valid results. It's possible there's a bug in vmstat or the underlying statistics it uses. I usually use gstat instead, but it might also be off (although my tests in the past have not proven that). Eric ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
At 04:38 AM 3/2/2007, O. Hartmann wrote: The last days I tried to figure out why some of my lab's FreeBSD boxes and also mine at home seem to be outperformed by some Linux setups around here and I saw something interesting. On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to Something strange about your setup I would say. I just tried on a Segate SATA drive off an ICH5 chipset (plain old P IV 2.4Ghz). Do you have an option in your BIOS for native mode or compatibility mode for the SATA controller ? If so, try toggling that to native SATA mode [ns4]% iostat -c 1000 tty ad4twed0 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 2 447 4.91 0 0.00 23.77 40 0.92 20 0 6 0 74 4 307 0.00 0 0.00 12.61 14 0.17 0 0 0 0 100 1 183 0.00 0 0.00 14.50 4 0.06 0 0 0 0 100 1 63 128.00 47 5.82 0.00 0 0.00 7 0 7 0 86 0 182 128.00 534 66.70 15.25 8 0.12 0 0 15 8 77 0 60 128.00 553 69.13 2.00 2 0.00 0 0 8 8 85 0 182 128.00 537 67.14 14.50 4 0.06 15 0 31 15 38 0 60 128.00 553 69.06 0.00 0 0.00 54 0 0 8 38 0 60 128.00 538 67.21 0.00 0 0.00 23 0 0 8 69 1 301 128.00 495 61.88 12.18 22 0.26 0 0 8 0 92 [ns4]# dd if=/dev/ad4 of=/dev/null bs=1024k ^C410+0 records in 410+0 records out 429916160 bytes transferred in 6.089321 secs (70601659 bytes/sec) [ns4]# [ns4]# atacontrol cap ad4 Protocol Serial ATA II device model ST3400833NS serial number 5NF25DTG firmware revision 3.AEH cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 781422768 sectors dma supported overlap not supported Feature Support EnableValue Vendor write cacheyes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management no no 65278/0xFEFE automatic acoustic management no no 0/0x00 254/0xFE [ns4]# ___ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: (S)ATA performance in FBSD 6.2/7.0
On Fri, 2 Mar 2007, Brooks Davis wrote: Also, you should time the actual copy and do the math to verify that vmstat is actually producing valid results. It's possible there's a bug in vmstat or the underlying statistics it uses. There is certainly a bug in the underlying statistics. For ATA disks, at least with the ata driver, the maximum transfer size in DMA mode is 64K, so any reports of a block size of 128K for SATA disks are wrong. The block size of 128K reported by vmstat is actually a virtual size. For most or types of disks, the GEOM layer virtualizes the physical maximum size MAXPHYS = 128K so that layers above GEOM including statistics gathering and file systems cannot see the physical size. For writing large files, this normally confuses ffs and vfs clustering into producing contiguous writes of 128K. This is good for efficiency, but it is not what the hardware sees or what you want for statistics. The contiguous writes of 128K get split up into 2 sequential writes of 64K. However, 64K is more than large enough for efficiency, so the bug in the underlying statistics doesn't matter, at least if vmstat reports only 128K blocks. If it reported 64K-blocks then you would have to worry about the contiguous block sizes being a mixture of 128K and much smaller blocks, with the much smaller blocks (actually, more the seeks across gaps to get to the smaller blocks) being very inefficient. Bruce ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
