RE: 5.8TB RAID5 SATA Array Questions - UPDATE
Question for the group...anyone know of a way to use an adaptec U160 controller to connect the NAS server to another system so the second system can write to the raid container thru the U160? (FBSD SATA RAID5 SERVER)--(GIG NETLINK)--(NETWORK) | -rw-(PCI U160)--(U160 CABLE)--(SERVER) -Original Message- From: Tomas Quintero [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 6:06 PM To: [EMAIL PROTECTED] Cc: Brent Wiese; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 5.8TB RAID5 SATA Array Questions - UPDATE
I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
Easy answer...the desktops are actually not windows based...they are Apple OSX / Linux systems...SMB is just for the transient Windows based systems that will need to access the array, but do not run NFS. -Original Message- From: Tomas Quintero [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 6:06 PM To: [EMAIL PROTECTED] Cc: Brent Wiese; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 5.8TB RAID5 SATA Array Questions - UPDATE
Ah my mistake, I hadn't read all of what was said in its entirety. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: Easy answer...the desktops are actually not windows based...they are Apple OSX / Linux systems...SMB is just for the transient Windows based systems that will need to access the array, but do not run NFS. -Original Message- From: Tomas Quintero [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 6:06 PM To: [EMAIL PROTECTED] Cc: Brent Wiese; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 5.8TB RAID5 SATA Array Questions - UPDATE
No. Doesn't work. Fdisk couldn't figure out how to partition it correctly. Actually it had a very hard time figuring out the correct Cylinder, Heads, Sectors values that worked correctly. I gave up on this. I boot from a 3Ware RAID5 host array (160GB). 2. No. I had 2.2TB arrays and I couldn't create a filesystem that big. I split them up in hardware to 1.1TB each and created 4 x 1.1TB arrays. No other workable solution I could find. Ben On 4/22/05, Edgar Martinez [EMAIL PROTECTED] wrote: Are you booting to the array? Is it over 2TB? Or are you mounting the array? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 5.8TB RAID5 SATA Array Questions - UPDATE
Hi Edgar, Good to hear you finally got it running. Sounds like you went through the same challenges I went through. I wound up getting FreeBSD 5.4-STABLE running and it's been stable for weeks. I put it through quite a bit of load lately and it seems to running well. Comments below: As much loved as BSD is to me.it simply just isn't up to the challenge at all.its far too difficult to get in a properly working state.and the limitations imposed are just too difficult to overcome easily. Sounds like you hit the same 2TB limit on both FBSD and Linux. What was the limitations that were too difficult to overcome? I ended up using Ubuntu which not only had all the driver support to all the devices and controllers.but also had little to no problem getting the system installed properly.It however does not like/want to boot to the array.so I installed additional drives (Seagate sata) and created a mirror (300GB) for the system to live on and bring up the array (/dev/md0) using mdadm.overall it was easy and nice.there are several caveats left to wrestle with. I wonder why it wouldn't want to boot off of your large array. It could be that it is way too big for the old PC bios to recognize. I think you could get around this by creating a small partition at the beginning of your array. I tried this too, but no luck. My arrays were over fiber channel but that should have been taken care of by the FC card. Currently although the 3ware controller can create a huge 4TB raid5 array, nothing exists that I am aware of that can utilize the entire container. Every single OS that exists seems to all share the 2TB limitations..so while the BIOS can see it.everything else will only see 2TB..this includes NFS on OSX (which don't get me started on the horrible implementation mistakes from apple and their poor NFS support..i mean NFSv4 comeon! Why is that hard!!) That is strange that OSX can't see larger than 2TB partitions over NFS. I would assume that an OSX client talking to an XServe would be able to see it. I haven't tested this so I wouldn't know for sure. I'm more curious about the 2TB limit on Linux. I figured Linux, with it's great file system support, would be able to handle a larger than 2TB partition. What were the limitations you ran into? So to get past Ubuntu's 2TB problem, I created 2xRAID5 2TB (1.8TB reporting) containers on the array.and then using software raid.created 1xRAID0 using the 2xRAID5 containers.which create 1xRAID0 @4TB. Why did software raid0 help you get over the 2TB limitation? Wouldn't it still appear as one filesystem that is way too big to use? Something doesn't add up here. Pun not intended. :) Utterly horrible.probably the WORST half-assed installation imaginable.in my honest opinion.here are my desires. I chose to break my 4.4TB system into 4 x 1.1TB arrays. This is very well supported by FreeBSD. The downside is that I had to modify my email system configuration and maintenance scripts to work with four smaller arrays rather than a single large one. I purposely avoided using software raid because it makes maintenance of the array a lot more complex. It usually doesn't take a lot of skills or time to fix a hardware array but the learning curve for fixing a software array is a lot higher. Plus I don't think software raid on linux is any good, or on FreeBSD for that matter. Create 1xRAID5 @ 4TB.install the OS TO the array.boot to the array and then share out 4TB via NFS/SMB.was that too much to ask?? Obviously it was. So in response.I can modified the requirements. Create [EMAIL PROTECTED] an OS TO a [EMAIL PROTECTED] to the RAID1..and SHARE out the 4TB. This is essentially what I did as well. Didn't know about the limitations when I first started. ben ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
Are you booting to the array? Is it over 2TB? Or are you mounting the array? -Original Message- From: Benson Wong [mailto:[EMAIL PROTECTED] Sent: Friday, April 22, 2005 1:40 PM To: [EMAIL PROTECTED] Cc: Nick Pavlica; Dan Nelson; Nick Evans; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE Hi Edgar, Good to hear you finally got it running. Sounds like you went through the same challenges I went through. I wound up getting FreeBSD 5.4-STABLE running and it's been stable for weeks. I put it through quite a bit of load lately and it seems to running well. Comments below: As much loved as BSD is to me.it simply just isn't up to the challenge at all.its far too difficult to get in a properly working state.and the limitations imposed are just too difficult to overcome easily. Sounds like you hit the same 2TB limit on both FBSD and Linux. What was the limitations that were too difficult to overcome? I ended up using Ubuntu which not only had all the driver support to all the devices and controllers.but also had little to no problem getting the system installed properly.It however does not like/want to boot to the array.so I installed additional drives (Seagate sata) and created a mirror (300GB) for the system to live on and bring up the array (/dev/md0) using mdadm.overall it was easy and nice.there are several caveats left to wrestle with. I wonder why it wouldn't want to boot off of your large array. It could be that it is way too big for the old PC bios to recognize. I think you could get around this by creating a small partition at the beginning of your array. I tried this too, but no luck. My arrays were over fiber channel but that should have been taken care of by the FC card. Currently although the 3ware controller can create a huge 4TB raid5 array, nothing exists that I am aware of that can utilize the entire container. Every single OS that exists seems to all share the 2TB limitations..so while the BIOS can see it.everything else will only see 2TB..this includes NFS on OSX (which don't get me started on the horrible implementation mistakes from apple and their poor NFS support..i mean NFSv4 comeon! Why is that hard!!) That is strange that OSX can't see larger than 2TB partitions over NFS. I would assume that an OSX client talking to an XServe would be able to see it. I haven't tested this so I wouldn't know for sure. I'm more curious about the 2TB limit on Linux. I figured Linux, with it's great file system support, would be able to handle a larger than 2TB partition. What were the limitations you ran into? So to get past Ubuntu's 2TB problem, I created 2xRAID5 2TB (1.8TB reporting) containers on the array.and then using software raid.created 1xRAID0 using the 2xRAID5 containers.which create 1xRAID0 @4TB. Why did software raid0 help you get over the 2TB limitation? Wouldn't it still appear as one filesystem that is way too big to use? Something doesn't add up here. Pun not intended. :) Utterly horrible.probably the WORST half-assed installation imaginable.in my honest opinion.here are my desires. I chose to break my 4.4TB system into 4 x 1.1TB arrays. This is very well supported by FreeBSD. The downside is that I had to modify my email system configuration and maintenance scripts to work with four smaller arrays rather than a single large one. I purposely avoided using software raid because it makes maintenance of the array a lot more complex. It usually doesn't take a lot of skills or time to fix a hardware array but the learning curve for fixing a software array is a lot higher. Plus I don't think software raid on linux is any good, or on FreeBSD for that matter. Create 1xRAID5 @ 4TB.install the OS TO the array.boot to the array and then share out 4TB via NFS/SMB.was that too much to ask?? Obviously it was. So in response.I can modified the requirements. Create [EMAIL PROTECTED] an OS TO a [EMAIL PROTECTED] to the RAID1..and SHARE out the 4TB. This is essentially what I did as well. Didn't know about the limitations when I first started. ben ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
All, So, after a soild chunk of life has fully been drained from me.here are several conclusions.obviously open for discussion if anyone wants to pick my brain.(yes we reduced our array size.you'll see why) As much loved as BSD is to me.it simply just isn't up to the challenge at all.its far too difficult to get in a properly working state.and the limitations imposed are just too difficult to overcome easily. I ended up using Ubuntu which not only had all the driver support to all the devices and controllers.but also had little to no problem getting the system installed properly.It however does not like/want to boot to the array.so I installed additional drives (Seagate sata) and created a mirror (300GB) for the system to live on and bring up the array (/dev/md0) using mdadm.overall it was easy and nice.there are several caveats left to wrestle with. Currently although the 3ware controller can create a huge 4TB raid5 array, nothing exists that I am aware of that can utilize the entire container. Every single OS that exists seems to all share the 2TB limitations..so while the BIOS can see it.everything else will only see 2TB..this includes NFS on OSX (which don't get me started on the horrible implementation mistakes from apple and their poor NFS support..i mean NFSv4 comeon! Why is that hard!!) So to get past Ubuntu's 2TB problem, I created 2xRAID5 2TB (1.8TB reporting) containers on the array.and then using software raid.created 1xRAID0 using the 2xRAID5 containers.which create 1xRAID0 @4TB. So.samba allows clients to see 2TB and FTP also allow for 2TB..this is probably the most complicated.and yet simple thing I can say I have done. Utterly horrible.probably the WORST half-assed installation imaginable.in my honest opinion.here are my desires. Create 1xRAID5 @ 4TB.install the OS TO the array.boot to the array and then share out 4TB via NFS/SMB.was that too much to ask?? Obviously it was. So in response.I can modified the requirements. Create [EMAIL PROTECTED] an OS TO a [EMAIL PROTECTED] to the RAID1..and SHARE out the 4TB. Any one else think they know of a better method?? _ From: Nick Pavlica [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 12:15 PM To: [EMAIL PROTECTED] Cc: Dan Nelson; Nick Evans; Benson Wong; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions On 4/15/05, Edgar Martinez [EMAIL PROTECTED] wrote: OK...so now we are going into some new territory...I am curious if you would care to elaborate a bit more...I am intrigued...if anyone wants me to do some experiments or test something, let me know...I for one welcome any attempts at pushing any limits or trying new things... I would help do some testing but I don't have any storage that large at the moment. I curious how 5.4RC2 or handles very large volumes. Have you already tried fdisk, newfs ? --Nick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
