Re: 3ware 9650 tips
On Mon, Jul 16, 2007 at 12:41:15PM +1000, David Chinner wrote: On Fri, Jul 13, 2007 at 03:36:46PM -0400, Justin Piszcz wrote: On Fri, 13 Jul 2007, Jon Collette wrote: Wouldn't Raid 6 be slower than Raid 5 because of the extra fault tolerance? http://www.enterprisenetworksandservers.com/monthly/art.php?1754 - 20% drop according to this article His 500GB WD drives are 7200RPM compared to the Raptors 10K. So his numbers will be slower. Justin what file system do you have running on the Raptors? I think thats an interesting point made by Joshua. I use XFS: When it comes to bandwidth, there is good reason for that. Trying to stick with a supported config as much as possible, I need to run ext3. As per usual, though, initial ext3 numbers are less than impressive. Using bonnie++ to get a baseline, I get (after doing 'blockdev --setra 65536' on the device): Write: 136MB/s Read: 384MB/s Proving it's not the hardware, with XFS the numbers look like: Write: 333MB/s Read: 465MB/s Those are pretty typical numbers. In my experience, ext3 is limited to about 250MB/s buffered write speed. It's not disk limited, it's design limited. e.g. on a disk subsystem where XFS was getting 4-5GB/s buffered write, ext3 was doing 250MB/s. http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-paper.pdf If you've got any sort of serious disk array, ext3 is not the filesystem to use To show what the difference is, I used blktrace and Chris Mason's seekwatcher script on a simple, single threaded dd command on a 12 disk dm RAID0 stripe: # dd if=/dev/zero of=/mnt/scratch/fred bs=1024k count=10k; sync http://oss.sgi.com/~dgc/writes/ext3_write.png http://oss.sgi.com/~dgc/writes/xfs_write.png You can see from the ext3 graph that it comes to a screeching halt every 5s (probably when pdflush runs) and at all other times the seek rate is 10,000 seeks/s. That's pretty bad for a brand new, empty filesystem and the only way it is sustained is the fact that the disks have their write caches turned on. ext4 will probably show better results, but I haven't got any of the tools installed to be able to test it The XFS pattern shows consistently an order of magnitude less seeks and consistent throughput above 600MB/s. To put the number of seeks in context, XFS is doing 512k I/Os at about 1200-1300 per second. The number of seeks? A bit above 10^3 per second or roughly 1 seek per I/O which is pretty much optimal. Cheers, Dave. -- Dave Chinner Principal Engineer SGI Australian Software Group - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Monday 16 July 2007 14:22:25 David Chinner wrote: You can see from the ext3 graph that it comes to a screeching halt every 5s (probably when pdflush runs) and at all other times the seek rate is 10,000 seeks/s. That's pretty bad for a brand new, empty filesystem and the only way it is sustained is the fact that the disks have their write caches turned on. ext4 will probably show better results, but I haven't got any of the tools installed to be able to test it I recently did some filesystem throuput tests, you may find it here http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/lustre/performance/ ldiskfs is ext3+extents+mballoc+some-smaller-patches, so is almost ext4 (delayed allocation is still missing, but the clusterfs/lustre people didn't port it and I'm afraid of hard to detect filesystem corruptions if I include it myself). Write performance is still slower than with xfs and I'm really considering to try to use xfs in lustre. Cheers, Bernd -- Bernd Schubert Q-Leap Networks GmbH - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Mon, 16 Jul 2007 at 12:41pm, David Chinner wrote If you've got any sort of serious disk array, ext3 is not the filesystem to use I do so wish that RedHat shared this view... -- Joshua Baker-LePain Department of Biomedical Engineering Duke University - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
David Chinner wrote: On Mon, Jul 16, 2007 at 12:41:15PM +1000, David Chinner wrote: On Fri, Jul 13, 2007 at 03:36:46PM -0400, Justin Piszcz wrote: ... If you've got any sort of serious disk array, ext3 is not the filesystem to use To show what the difference is, I used blktrace and Chris Mason's seekwatcher script on a simple, single threaded dd command on a 12 disk dm RAID0 stripe: # dd if=/dev/zero of=/mnt/scratch/fred bs=1024k count=10k; sync http://oss.sgi.com/~dgc/writes/ext3_write.png http://oss.sgi.com/~dgc/writes/xfs_write.png Were those all with default mkfs mount options? ext3 in writeback mode might be an interesting comparison too. -Eric - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Advocacy] Re: 3ware 9650 tips
On Mon, Jul 16, 2007 at 08:40:00PM +0300, Al Boldi wrote: XFS surely rocks, but it's missing one critical component: data=ordered And that's one component that's just too critical to overlook for an enterprise environment that is built on data-integrity over performance. So that's the secret why people still use ext3, and XFS' reliance on external hardware to ensure integrity is really misplaced. Now, maybe when we get the data=ordered onto the VFS level, then maybe XFS may become viable for the enterprise, and ext3 may cease to be KING. Wow, thanks for bringing an advocacy thread onto linux-fsdevel. Just what we wanted. Do you have any insight into how to get the data=ordered onto the VFS level? Because to me, that sounds like pure nonsense. -- Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Mon, Jul 16, 2007 at 11:43:24AM -0400, Joshua Baker-LePain wrote: On Mon, 16 Jul 2007 at 12:41pm, David Chinner wrote If you've got any sort of serious disk array, ext3 is not the filesystem to use I do so wish that RedHat shared this view... So they support XFS in Fedora, but not in RHEL?? (I've been using Fedora...) - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC] VFS: data=ordered (was: [Advocacy] Re: 3ware 9650 tips)
Matthew Wilcox wrote: On Mon, Jul 16, 2007 at 08:40:00PM +0300, Al Boldi wrote: XFS surely rocks, but it's missing one critical component: data=ordered And that's one component that's just too critical to overlook for an enterprise environment that is built on data-integrity over performance. So that's the secret why people still use ext3, and XFS' reliance on external hardware to ensure integrity is really misplaced. Now, maybe when we get the data=ordered onto the VFS level, then maybe XFS may become viable for the enterprise, and ext3 may cease to be KING. Wow, thanks for bringing an advocacy thread onto linux-fsdevel. Just what we wanted. Do you have any insight into how to get the data=ordered onto the VFS level? Because to me, that sounds like pure nonsense. Well, conceptually it sounds like a piece of cake, technically your guess is as good as mine. IIRC, akpm once mentioned something like this. But seriously, can you think of a technical reason why it shouldn't be possible to abstract data=ordered mode out into the VFS? Thanks! -- Al - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Advocacy] Re: 3ware 9650 tips
On Mon, 2007-07-16 at 11:48 -0600, Matthew Wilcox wrote: Wow, thanks for bringing an advocacy thread onto linux-fsdevel. Just what we wanted. Do you have any insight into how to get the data=ordered onto the VFS level? Because to me, that sounds like pure nonsense. First off, I have no idea who decided to respond to my post and CC: linux-fsdevel on it. In retrospect, secondly, I should have not posted my post to linux-raid in the first place (is that list now mirrored to linux-fsdevel or something?). I was just sharing in my frustration of the lack of XFS support by Red Hat. So, lastly and in any case, my apologies to all, even if I did not proliferate it to linux-fsdevel, it was probably not ideal for me to post such to anything on vger.kernel.org (like linux-raid) in the first place. -- Bryan J. Smith Professional, Technical Annoyance mailto:[EMAIL PROTECTED] http://thebs413.blogspot.com Fission Power: An Inconvenient Solution - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Mon, Jul 16, 2007 at 10:50:34AM -0500, Eric Sandeen wrote: David Chinner wrote: On Mon, Jul 16, 2007 at 12:41:15PM +1000, David Chinner wrote: On Fri, Jul 13, 2007 at 03:36:46PM -0400, Justin Piszcz wrote: ... If you've got any sort of serious disk array, ext3 is not the filesystem to use To show what the difference is, I used blktrace and Chris Mason's seekwatcher script on a simple, single threaded dd command on a 12 disk dm RAID0 stripe: # dd if=/dev/zero of=/mnt/scratch/fred bs=1024k count=10k; sync http://oss.sgi.com/~dgc/writes/ext3_write.png http://oss.sgi.com/~dgc/writes/xfs_write.png Were those all with default mkfs mount options? ext3 in writeback mode might be an interesting comparison too. Defaults. i.e. # mkfs.ext3 /dev/mapper/dm0 # mkfs.xfs /dev/mapper/dm0 The mkfs.xfs picked up sunit/swidth correctly from the dm volume. Last time I checked, writeback made little difference to ext3 throughput; maybe 5-10% at most. I'll run it again later today... Cheers, Dave. -- Dave Chinner Principal Engineer SGI Australian Software Group - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Fri, Jul 13, 2007 at 03:36:46PM -0400, Justin Piszcz wrote: On Fri, 13 Jul 2007, Jon Collette wrote: Wouldn't Raid 6 be slower than Raid 5 because of the extra fault tolerance? http://www.enterprisenetworksandservers.com/monthly/art.php?1754 - 20% drop according to this article His 500GB WD drives are 7200RPM compared to the Raptors 10K. So his numbers will be slower. Justin what file system do you have running on the Raptors? I think thats an interesting point made by Joshua. I use XFS: When it comes to bandwidth, there is good reason for that. Trying to stick with a supported config as much as possible, I need to run ext3. As per usual, though, initial ext3 numbers are less than impressive. Using bonnie++ to get a baseline, I get (after doing 'blockdev --setra 65536' on the device): Write: 136MB/s Read: 384MB/s Proving it's not the hardware, with XFS the numbers look like: Write: 333MB/s Read: 465MB/s Those are pretty typical numbers. In my experience, ext3 is limited to about 250MB/s buffered write speed. It's not disk limited, it's design limited. e.g. on a disk subsystem where XFS was getting 4-5GB/s buffered write, ext3 was doing 250MB/s. http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-paper.pdf If you've got any sort of serious disk array, ext3 is not the filesystem to use How many folks are using these? Any tuning tips? Make sure you tell XFS the correct sunit/swidth. For hardware raid5/6, sunit = per-disk chunksize, swidth = number of *data* disks in array. Cheers, Dave. -- Dave Chinner Principal Engineer SGI Australian Software Group - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Fri, 13 Jul 2007, Andrew Klaassen wrote: --- Justin Piszcz [EMAIL PROTECTED] wrote: To give you an example I get 464MB/s write and 627MB/s with a 10 disk raptor software raid5. Is that with the 9650? Andrew Sorry no, its with software raid 5 and the 965 chipset + three SATA PCI-e cards. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Fri, 13 Jul 2007, Justin Piszcz wrote: You are using HW RAID then? Those numbers seem pretty awful for that setup, including linux-raid@ even it though it appears you're running HW raid, this is rather peculiar. No, it has been discussed numerous times on this list. SW raid is faster because it has access to (often) gigabytes of block cache, which the HW raid controller doesn't have. SW raid is therefore able to avoid a lot of reads when it needs to write, speeding things up considerably. I always use 3ware HW-raid though as I consider it more reliable. Since most of my access is write once, read many write speed isn't as important to me as data integrity. Take your 3ware HW-raid, do a dd (read or write) to the device and see it being very quick (because it can fit all the data into its cache as it either reads or writes), then put a filesystem on it and do writes there, especially journaled writes, and see write speed go down to 1/10 or so. -- Mikael Abrahamssonemail: [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
--- Justin Piszcz [EMAIL PROTECTED] wrote: On Fri, 13 Jul 2007, Andrew Klaassen wrote: --- Justin Piszcz [EMAIL PROTECTED] wrote: To give you an example I get 464MB/s write and 627MB/s with a 10 disk raptor software raid5. Is that with the 9650? Andrew Sorry no, its with software raid 5 and the 965 chipset + three SATA PCI-e cards. Which cards? Those are pretty good numbers, so I'm interested. Andrew Never miss an email again! Yahoo! Toolbar alerts you the instant new Mail arrives. http://tools.search.yahoo.com/toolbar/features/mail/ - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Sat, 14 Jul 2007, Andrew Klaassen wrote: --- Justin Piszcz [EMAIL PROTECTED] wrote: On Fri, 13 Jul 2007, Andrew Klaassen wrote: --- Justin Piszcz [EMAIL PROTECTED] wrote: To give you an example I get 464MB/s write and 627MB/s with a 10 disk raptor software raid5. Is that with the 9650? Andrew Sorry no, its with software raid 5 and the 965 chipset + three SATA PCI-e cards. Which cards? Those are pretty good numbers, so I'm interested. Andrew 03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01) $19.99 2 port SYBA cards (Silicon Image 3132s) http://www.directron.com/sdsa2pex2ir.html - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
--- Mikael Abrahamsson [EMAIL PROTECTED] wrote: Take your 3ware HW-raid, do a dd (read or write) to the device and see it being very quick (because it can fit all the data into its cache as it either reads or writes), then put a filesystem on it and do writes there, especially journaled writes, and see write speed go down to 1/10 or so. How does non-cached performance tend to compare? Andrew Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase. http://farechase.yahoo.com/ - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
--- Justin Piszcz [EMAIL PROTECTED] wrote: 03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01) $19.99 2 port SYBA cards (Silicon Image 3132s) http://www.directron.com/sdsa2pex2ir.html Cool, thanks. What are your bonnie++ rewrite numbers? Andrew Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out. http://answers.yahoo.com/dir/?link=listsid=396545469 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Sat, 14 Jul 2007, Andrew Klaassen wrote: --- Justin Piszcz [EMAIL PROTECTED] wrote: 03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01) $19.99 2 port SYBA cards (Silicon Image 3132s) http://www.directron.com/sdsa2pex2ir.html Cool, thanks. What are your bonnie++ rewrite numbers? Andrew 3 runs: p34,16G,77169,99,413276,80,155348,26,78932,99,535482,41,607.0,0,16:10:16/64,1500,12,4886,15,1790,16,1821,17,6081,19,2159,19 p34,16G,77659,99,451593,87,167267,28,79058,99,584310,45,613.1,0,16:10:16/64,1843,15,6006,31,1325,11,1204,12,3629,12,3324,31 p34,16G,77873,99,441881,87,166384,28,75182,99,566384,43,619.4,0,16:10:16/64,1537,13,4474,15,1827,18,880,8,7658,22,3864,36 avg: p34,16G,77567,99,435583,84.6667,163000,27.,77724,99,562059,43,613.167,0,16:10:16/64,1626.67,13.,5122,20.,1647.33,15,1301.67,12.,5789.33,17.6667,3115.67,28.6667 rewrite: 163000 KiB/s When tarring 4.4GB (of backup files) it takes about 20 seconds on XFS. Seems to vary as I change my configuration a lot: Here is from a while back: Version 1.03 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP p34-raid515696M 73755 99 411445 75 198639 34 78721 99 483169 39 584.8 0 --Sequential Create-- Random Create p34-raid5,15696M,73755,99,411445,75,198639,34,78721,99,483169,39,584.8,0,16:10:16/64,919,8,9940,28,2841,18,922,8,3225,10,2422,18 Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
Wouldn't Raid 6 be slower than Raid 5 because of the extra fault tolerance? http://www.enterprisenetworksandservers.com/monthly/art.php?1754 - 20% drop according to this article His 500GB WD drives are 7200RPM compared to the Raptors 10K. So his numbers will be slower. Justin what file system do you have running on the Raptors? I think thats an interesting point made by Joshua. Justin Piszcz wrote: On Fri, 13 Jul 2007, Joshua Baker-LePain wrote: My new system has a 3ware 9650SE-24M8 controller hooked to 24 500GB WD drives. The controller is set up as a RAID6 w/ a hot spare. OS is CentOS 5 x86_64. It's all running on a couple of Xeon 5130s on a Supermicro X7DBE motherboard w/ 4GB of RAM. Trying to stick with a supported config as much as possible, I need to run ext3. As per usual, though, initial ext3 numbers are less than impressive. Using bonnie++ to get a baseline, I get (after doing 'blockdev --setra 65536' on the device): Write: 136MB/s Read: 384MB/s Proving it's not the hardware, with XFS the numbers look like: Write: 333MB/s Read: 465MB/s How many folks are using these? Any tuning tips? Thanks. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University Let's try that again with the right address :) You are using HW RAID then? Those numbers seem pretty awful for that setup, including linux-raid@ even it though it appears you're running HW raid, this is rather peculiar. To give you an example I get 464MB/s write and 627MB/s with a 10 disk raptor software raid5. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Fri, 13 Jul 2007, Joshua Baker-LePain wrote: My new system has a 3ware 9650SE-24M8 controller hooked to 24 500GB WD drives. The controller is set up as a RAID6 w/ a hot spare. OS is CentOS 5 x86_64. It's all running on a couple of Xeon 5130s on a Supermicro X7DBE motherboard w/ 4GB of RAM. Trying to stick with a supported config as much as possible, I need to run ext3. As per usual, though, initial ext3 numbers are less than impressive. Using bonnie++ to get a baseline, I get (after doing 'blockdev --setra 65536' on the device): Write: 136MB/s Read: 384MB/s Proving it's not the hardware, with XFS the numbers look like: Write: 333MB/s Read: 465MB/s How many folks are using these? Any tuning tips? Thanks. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University Let's try that again with the right address :) You are using HW RAID then? Those numbers seem pretty awful for that setup, including linux-raid@ even it though it appears you're running HW raid, this is rather peculiar. To give you an example I get 464MB/s write and 627MB/s with a 10 disk raptor software raid5. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
On Fri, 13 Jul 2007 at 2:35pm, Justin Piszcz wrote On Fri, 13 Jul 2007, Joshua Baker-LePain wrote: My new system has a 3ware 9650SE-24M8 controller hooked to 24 500GB WD drives. The controller is set up as a RAID6 w/ a hot spare. OS is CentOS 5 x86_64. It's all running on a couple of Xeon 5130s on a Supermicro X7DBE motherboard w/ 4GB of RAM. Trying to stick with a supported config as much as possible, I need to run ext3. As per usual, though, initial ext3 numbers are less than impressive. Using bonnie++ to get a baseline, I get (after doing 'blockdev --setra 65536' on the device): Write: 136MB/s Read: 384MB/s Proving it's not the hardware, with XFS the numbers look like: Write: 333MB/s Read: 465MB/s How many folks are using these? Any tuning tips? Thanks. You are using HW RAID then? Those numbers seem pretty awful for that setup, including linux-raid@ even it though it appears you're running HW raid, this is rather peculiar. Yep, hardware RAID -- I need the hot swappability (which, AFAIK, is still an issue with md). -- Joshua Baker-LePain Department of Biomedical Engineering Duke University - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
Joshua Baker-LePain wrote: [] Yep, hardware RAID -- I need the hot swappability (which, AFAIK, is still an issue with md). Just out of curiocity - what do you mean by swappability ? For many years we're using linux software raid, we had no problems with swappability of the component drives (in case of drive failures and what not). With non-hotswappable drives (old scsi and ide ones), rebooting is needed for the system to recognize the drives. For modern sas/sata drives, i can replace a faulty drive without anyone noticing... Maybe you're referring to something else? Thanks. /mjt - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 3ware 9650 tips
--- Justin Piszcz [EMAIL PROTECTED] wrote: To give you an example I get 464MB/s write and 627MB/s with a 10 disk raptor software raid5. Is that with the 9650? Andrew Fussy? Opinionated? Impossible to please? Perfect. Join Yahoo!'s user panel and lay it on us. http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
