Re: Raid over 48 disks ... for real now

2008-01-18 Thread michael
Quoting Norman Elton [EMAIL PROTECTED]: I posed the question a few weeks ago about how to best accommodate software RAID over an array of 48 disks (a Sun X4500 server, a.k.a. Thumper). I appreciate all the suggestions. Well, the hardware is here. It is indeed six Marvell 88SX6081 SATA

Re: Raid over 48 disks ... for real now

2008-01-18 Thread Greg Cormier
I wonder how long it would take to run an fsck on one large filesystem? :) I would imagine you'd have time to order a new system, build it, and restore the backups before the fsck was done! - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL

Re: Raid over 48 disks ... for real now

2008-01-18 Thread Norman Elton
It is quite a box. There's a picture of the box with the cover removed on Sun's website: http://www.sun.com/images/k3/k3_sunfirex4500_4.jpg From the X4500 homepage, there's a gallery of additional pictures. The drives drop in from the top. Massive fans channel air in the small gaps between the

Re: Raid over 48 disks ... for real now

2008-01-18 Thread Jon Lewis
On Thu, 17 Jan 2008, Janek Kozicki wrote: I wish RHEL would support XFS/ZFS, but for now, I'm stuck with ext3. there is ext4 (or ext4dev) - it's an ext3 modified to support 1024 PB size (1048576 TB). You could check if it's feasible. Personally I'd always stick with ext2/ext3/ext4 since it is

Re: Raid over 48 disks ... for real now

2008-01-17 Thread Norman Elton
Hi, sounds like a monster server. I am interested in how you will make the space useful to remote machines- iscsi? this is what I am researching currently. Yes, it's a honker of a box. It will be collecting data from various collector servers. The plan right now is to collect the file to

Re: Raid over 48 disks

2007-12-25 Thread pg_mh
On Wed, 19 Dec 2007 07:28:20 +1100, Neil Brown [EMAIL PROTECTED] said: [ ... what to do with 48 drive Sun Thumpers ... ] neilb I wouldn't create a raid5 or raid6 on all 48 devices. neilb RAID5 only survives a single device failure and with that neilb many devices, the chance of a second

Re: Raid over 48 disks

2007-12-25 Thread Bill Davidsen
Peter Grandi wrote: On Wed, 19 Dec 2007 07:28:20 +1100, Neil Brown [EMAIL PROTECTED] said: [ ... what to do with 48 drive Sun Thumpers ... ] neilb I wouldn't create a raid5 or raid6 on all 48 devices. neilb RAID5 only survives a single device failure and with that neilb many devices,

Re: Raid over 48 disks

2007-12-21 Thread Leif Nixon
Norman Elton [EMAIL PROTECTED] writes: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ I think BNL's evalation of Solaris/ZFS vs. Linux/MD on a thumper might be of interest:

Re: Raid over 48 disks

2007-12-21 Thread Leif Nixon
Mattias Wadenstein [EMAIL PROTECTED] writes: There are those that have run Linux MD RAID on thumpers before. I vaguely recall some driver issues (unrelated to MD) that made it less suitable than solaris, but that might be fixed in recent kernels. I think that was mainly an issue for people

Re: Raid over 48 disks

2007-12-20 Thread Thiemo Nagel
Bill Davidsen wrote: 16k read64k write chunk sizeRAID 5RAID 6RAID 5RAID 6 128k492497268270 256k615530288270 512k625607230174 1024k 65062017075 What is your stripe cache size? I didn't fiddle

Re: Raid over 48 disks

2007-12-20 Thread Bill Davidsen
Thiemo Nagel wrote: Bill Davidsen wrote: 16k read64k write chunk sizeRAID 5RAID 6RAID 5RAID 6 128k492497268270 256k615530288270 512k625607230174 1024k 65062017075 What is your stripe cache

Re: Raid over 48 disks

2007-12-19 Thread Mattias Wadenstein
On Wed, 19 Dec 2007, Neil Brown wrote: On Tuesday December 18, [EMAIL PROTECTED] wrote: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ Basically, it's a server with 48 SATA hard drives. No hardware RAID.

Re: Raid over 48 disks

2007-12-19 Thread Russell Smith
Guy Watkins wrote: } -Original Message- } From: [EMAIL PROTECTED] [mailto:linux-raid- } [EMAIL PROTECTED] On Behalf Of Brendan Conoboy } Sent: Tuesday, December 18, 2007 3:36 PM } To: Norman Elton } Cc: linux-raid@vger.kernel.org } Subject: Re: Raid over 48 disks } } Norman Elton wrote

Re: Raid over 48 disks

2007-12-19 Thread Bill Davidsen
Thiemo Nagel wrote: Performance of the raw device is fair: # dd if=/dev/md2 of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied, 15.6071 seconds, 550 MB/s Somewhat less through ext3 (created with -E stride=64): # dd if=largetestfile of=/dev/zero bs=128k count=64k 8589934592 bytes

Re: Raid over 48 disks

2007-12-19 Thread Justin Piszcz
On Wed, 19 Dec 2007, Bill Davidsen wrote: Thiemo Nagel wrote: Performance of the raw device is fair: # dd if=/dev/md2 of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied, 15.6071 seconds, 550 MB/s Somewhat less through ext3 (created with -E stride=64): # dd if=largetestfile

Re: Raid over 48 disks

2007-12-19 Thread Bill Davidsen
Mattias Wadenstein wrote: On Wed, 19 Dec 2007, Neil Brown wrote: On Tuesday December 18, [EMAIL PROTECTED] wrote: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ Basically, it's a server with 48 SATA hard

Re: Raid over 48 disks

2007-12-18 Thread Justin Piszcz
On Tue, 18 Dec 2007, Norman Elton wrote: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ Basically, it's a server with 48 SATA hard drives. No hardware RAID. It's designed for Sun's ZFS filesystem.

Re: Raid over 48 disks

2007-12-18 Thread Robin Hill
On Tue Dec 18, 2007 at 12:29:27PM -0500, Norman Elton wrote: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ Basically, it's a server with 48 SATA hard drives. No hardware RAID. It's designed for Sun's

Re: Raid over 48 disks

2007-12-18 Thread Thiemo Nagel
Dear Norman, So... we're curious how Linux will handle such a beast. Has anyone run MD software RAID over so many disks? Then piled LVM/ext3 on top of that? Any suggestions? Are we crazy to think this is even possible? I'm running 22x 500GB disks attached to RocketRaid2340 and NFORCE-MCP55

Re: Raid over 48 disks

2007-12-18 Thread Norman Elton
Thiemo -- I'm not familiar with RocketRaid. Is it handling the RAID for you, or are you using MD? Thanks, all, for your feedback! I'm still surprised nobody has tried this on one of these Sun boxes yet. I've signed up for some demo hardware. I'll post what I find. Norman On Dec 18,

Re: Raid over 48 disks

2007-12-18 Thread Thiemo Nagel
Dear Norman, I'm not familiar with RocketRaid. Is it handling the RAID for you, or are you using MD? I'm using md. The controller is in a mode that exports all drives individually. Kind regards, Thiemo - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a

Re: Raid over 48 disks

2007-12-18 Thread Justin Piszcz
On Tue, 18 Dec 2007, Thiemo Nagel wrote: Dear Norman, So... we're curious how Linux will handle such a beast. Has anyone run MD software RAID over so many disks? Then piled LVM/ext3 on top of that? Any suggestions? Are we crazy to think this is even possible? I'm running 22x 500GB

Re: Raid over 48 disks

2007-12-18 Thread Brendan Conoboy
Norman Elton wrote: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ Neat- 6 8 port SATA controllers! It'll be worth checking to be sure each controller has equal bandwidth. If some controllers are on

Re: Raid over 48 disks

2007-12-18 Thread Neil Brown
On Tuesday December 18, [EMAIL PROTECTED] wrote: We're investigating the possibility of running Linux (RHEL) on top of Sun's X4500 Thumper box: http://www.sun.com/servers/x64/x4500/ Basically, it's a server with 48 SATA hard drives. No hardware RAID. It's designed for Sun's ZFS

Re: Raid over 48 disks

2007-12-18 Thread Thiemo Nagel
Performance of the raw device is fair: # dd if=/dev/md2 of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied, 15.6071 seconds, 550 MB/s Somewhat less through ext3 (created with -E stride=64): # dd if=largetestfile of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied,

Re: Raid over 48 disks

2007-12-18 Thread Jon Nelson
On 12/18/07, Thiemo Nagel [EMAIL PROTECTED] wrote: Performance of the raw device is fair: # dd if=/dev/md2 of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied, 15.6071 seconds, 550 MB/s Somewhat less through ext3 (created with -E stride=64): # dd if=largetestfile

Re: Raid over 48 disks

2007-12-18 Thread Justin Piszcz
On Tue, 18 Dec 2007, Thiemo Nagel wrote: Performance of the raw device is fair: # dd if=/dev/md2 of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied, 15.6071 seconds, 550 MB/s Somewhat less through ext3 (created with -E stride=64): # dd if=largetestfile of=/dev/zero bs=128k

Re: Raid over 48 disks

2007-12-18 Thread Thiemo Nagel
16k read64k write chunk sizeRAID 5 RAID 6 RAID 5 RAID 6 128k492 497 268 270 256k615 530 288 270 512k625 607 230 174 1024k 650 620 170 75 It strikes me that these numbers are meaningless without knowing if that

Re: Raid over 48 disks

2007-12-18 Thread Justin Piszcz
On Tue, 18 Dec 2007, Jon Nelson wrote: On 12/18/07, Thiemo Nagel [EMAIL PROTECTED] wrote: Performance of the raw device is fair: # dd if=/dev/md2 of=/dev/zero bs=128k count=64k 8589934592 bytes (8.6 GB) copied, 15.6071 seconds, 550 MB/s Somewhat less through ext3 (created with -E

RE: Raid over 48 disks

2007-12-18 Thread Guy Watkins
} -Original Message- } From: [EMAIL PROTECTED] [mailto:linux-raid- } [EMAIL PROTECTED] On Behalf Of Brendan Conoboy } Sent: Tuesday, December 18, 2007 3:36 PM } To: Norman Elton } Cc: linux-raid@vger.kernel.org } Subject: Re: Raid over 48 disks } } Norman Elton wrote: } We're

RE: Raid over 48 disks

2007-12-18 Thread Justin Piszcz
On Tue, 18 Dec 2007, Guy Watkins wrote: } -Original Message- } From: [EMAIL PROTECTED] [mailto:linux-raid- } [EMAIL PROTECTED] On Behalf Of Brendan Conoboy } Sent: Tuesday, December 18, 2007 3:36 PM } To: Norman Elton } Cc: linux-raid@vger.kernel.org } Subject: Re: Raid over 48 disks

RE: Raid over 48 disks

2007-12-18 Thread Justin Piszcz
@vger.kernel.org } Subject: Re: Raid over 48 disks } } Norman Elton wrote: } We're investigating the possibility of running Linux (RHEL) on top of } Sun's X4500 Thumper box: } } http://www.sun.com/servers/x64/x4500/ } } Neat- 6 8 port SATA controllers! It'll be worth checking to be sure } each