5.8TB RAID5 SATA Array - CONGRATS!
All, I would like to THANK the dev team on 5.4 for all their hardwork. It has been delayed but I have to say that it really feels SOLID, and works like a dream...they have also surmounted some pretty HUGE problems in big disk supportKUDOS THANKS AWESOME...guys 5.4 is WORTH the wait... I think I can be the first to say that I have successfully created the worlds first sata based 16TB RAID container, on a single NAS server...tastes good... YOU GUYS ROCK! -Original Message- From: Tomas Quintero [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 6:06 PM To: [EMAIL PROTECTED] Cc: Brent Wiese; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
Question for the group...anyone know of a way to use an adaptec U160 controller to connect the NAS server to another system so the second system can write to the raid container thru the U160? (FBSD SATA RAID5 SERVER)--(GIG NETLINK)--(NETWORK) | -rw-(PCI U160)--(U160 CABLE)--(SERVER) -Original Message- From: Tomas Quintero [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 6:06 PM To: [EMAIL PROTECTED] Cc: Brent Wiese; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
Easy answer...the desktops are actually not windows based...they are Apple OSX / Linux systems...SMB is just for the transient Windows based systems that will need to access the array, but do not run NFS. -Original Message- From: Tomas Quintero [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 6:06 PM To: [EMAIL PROTECTED] Cc: Brent Wiese; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE I am almost a bit curious why you didn't go with a Microsoft based solution in a situation like this, where you are needing to provide SMB based file sharing to obviously Windows client desktops. Another solution would be to setup a dedicated NAS of some sort. But I suppose it's too late for all of that. On 4/25/05, Edgar Martinez [EMAIL PROTECTED] wrote: No flaming here, when dealing with projects this big, you cannot be bias obviously because generally it is someone else's time and money that is on the line. Thanks for the info, I didn't know the whole second array thing, that would explain some of the weirdness that I have been seeing. -Original Message- From: Brent Wiese [mailto:[EMAIL PROTECTED] Sent: Monday, April 25, 2005 12:54 PM To: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: RE: 5.8TB RAID5 SATA Array Questions - UPDATE Any one else think they know of a better method?? Well, I'm probably going to get totally flamed for this, but since you asked... The better method is to install Windows 2003 Server. Assemble your drives into 2TB or less RAID5 volumes (btw, you only want 1 per 3Ware card, more on that in a second) and use Windows 2003 to span those volumes. It'll show up as one drive after that. There is some limit, but I can't remember what it is. Its huge though. And in case you didn't know, 3Ware cards are only speed-optimized for the first array. Subsequent arrays on a card run painfully slow. They won't say it in any of their lit, but if you corner their support people, they'll admit it (it obvious if you try it). Sorry to mention M$ here, but it sounds like you invested incredible amounts of time, and even Windows 2003 can be cheaper than your time at some point. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] -- -Tomas Quintero ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
Are you booting to the array? Is it over 2TB? Or are you mounting the array? -Original Message- From: Benson Wong [mailto:[EMAIL PROTECTED] Sent: Friday, April 22, 2005 1:40 PM To: [EMAIL PROTECTED] Cc: Nick Pavlica; Dan Nelson; Nick Evans; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions - UPDATE Hi Edgar, Good to hear you finally got it running. Sounds like you went through the same challenges I went through. I wound up getting FreeBSD 5.4-STABLE running and it's been stable for weeks. I put it through quite a bit of load lately and it seems to running well. Comments below: As much loved as BSD is to me.it simply just isn't up to the challenge at all.its far too difficult to get in a properly working state.and the limitations imposed are just too difficult to overcome easily. Sounds like you hit the same 2TB limit on both FBSD and Linux. What was the limitations that were too difficult to overcome? I ended up using Ubuntu which not only had all the driver support to all the devices and controllers.but also had little to no problem getting the system installed properly.It however does not like/want to boot to the array.so I installed additional drives (Seagate sata) and created a mirror (300GB) for the system to live on and bring up the array (/dev/md0) using mdadm.overall it was easy and nice.there are several caveats left to wrestle with. I wonder why it wouldn't want to boot off of your large array. It could be that it is way too big for the old PC bios to recognize. I think you could get around this by creating a small partition at the beginning of your array. I tried this too, but no luck. My arrays were over fiber channel but that should have been taken care of by the FC card. Currently although the 3ware controller can create a huge 4TB raid5 array, nothing exists that I am aware of that can utilize the entire container. Every single OS that exists seems to all share the 2TB limitations..so while the BIOS can see it.everything else will only see 2TB..this includes NFS on OSX (which don't get me started on the horrible implementation mistakes from apple and their poor NFS support..i mean NFSv4 comeon! Why is that hard!!) That is strange that OSX can't see larger than 2TB partitions over NFS. I would assume that an OSX client talking to an XServe would be able to see it. I haven't tested this so I wouldn't know for sure. I'm more curious about the 2TB limit on Linux. I figured Linux, with it's great file system support, would be able to handle a larger than 2TB partition. What were the limitations you ran into? So to get past Ubuntu's 2TB problem, I created 2xRAID5 2TB (1.8TB reporting) containers on the array.and then using software raid.created 1xRAID0 using the 2xRAID5 containers.which create 1xRAID0 @4TB. Why did software raid0 help you get over the 2TB limitation? Wouldn't it still appear as one filesystem that is way too big to use? Something doesn't add up here. Pun not intended. :) Utterly horrible.probably the WORST half-assed installation imaginable.in my honest opinion.here are my desires. I chose to break my 4.4TB system into 4 x 1.1TB arrays. This is very well supported by FreeBSD. The downside is that I had to modify my email system configuration and maintenance scripts to work with four smaller arrays rather than a single large one. I purposely avoided using software raid because it makes maintenance of the array a lot more complex. It usually doesn't take a lot of skills or time to fix a hardware array but the learning curve for fixing a software array is a lot higher. Plus I don't think software raid on linux is any good, or on FreeBSD for that matter. Create 1xRAID5 @ 4TB.install the OS TO the array.boot to the array and then share out 4TB via NFS/SMB.was that too much to ask?? Obviously it was. So in response.I can modified the requirements. Create [EMAIL PROTECTED] an OS TO a [EMAIL PROTECTED] to the RAID1..and SHARE out the 4TB. This is essentially what I did as well. Didn't know about the limitations when I first started. ben ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions - UPDATE
All, So, after a soild chunk of life has fully been drained from me.here are several conclusions.obviously open for discussion if anyone wants to pick my brain.(yes we reduced our array size.you'll see why) As much loved as BSD is to me.it simply just isn't up to the challenge at all.its far too difficult to get in a properly working state.and the limitations imposed are just too difficult to overcome easily. I ended up using Ubuntu which not only had all the driver support to all the devices and controllers.but also had little to no problem getting the system installed properly.It however does not like/want to boot to the array.so I installed additional drives (Seagate sata) and created a mirror (300GB) for the system to live on and bring up the array (/dev/md0) using mdadm.overall it was easy and nice.there are several caveats left to wrestle with. Currently although the 3ware controller can create a huge 4TB raid5 array, nothing exists that I am aware of that can utilize the entire container. Every single OS that exists seems to all share the 2TB limitations..so while the BIOS can see it.everything else will only see 2TB..this includes NFS on OSX (which don't get me started on the horrible implementation mistakes from apple and their poor NFS support..i mean NFSv4 comeon! Why is that hard!!) So to get past Ubuntu's 2TB problem, I created 2xRAID5 2TB (1.8TB reporting) containers on the array.and then using software raid.created 1xRAID0 using the 2xRAID5 containers.which create 1xRAID0 @4TB. So.samba allows clients to see 2TB and FTP also allow for 2TB..this is probably the most complicated.and yet simple thing I can say I have done. Utterly horrible.probably the WORST half-assed installation imaginable.in my honest opinion.here are my desires. Create 1xRAID5 @ 4TB.install the OS TO the array.boot to the array and then share out 4TB via NFS/SMB.was that too much to ask?? Obviously it was. So in response.I can modified the requirements. Create [EMAIL PROTECTED] an OS TO a [EMAIL PROTECTED] to the RAID1..and SHARE out the 4TB. Any one else think they know of a better method?? _ From: Nick Pavlica [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 12:15 PM To: [EMAIL PROTECTED] Cc: Dan Nelson; Nick Evans; Benson Wong; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions On 4/15/05, Edgar Martinez [EMAIL PROTECTED] wrote: OK...so now we are going into some new territory...I am curious if you would care to elaborate a bit more...I am intrigued...if anyone wants me to do some experiments or test something, let me know...I for one welcome any attempts at pushing any limits or trying new things... I would help do some testing but I don't have any storage that large at the moment. I curious how 5.4RC2 or handles very large volumes. Have you already tried fdisk, newfs ? --Nick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
Yeah it was pretty much boo hoo hoo...it appears we have either backplane, MI cable issues, or controller problems...I was only getting 11 drives available with improper identification...so I am going thru the tedious task of ripping it all down, and testing backplanes, drives, and cables...one at a time... On a side note...I was able to do my bastardization procedure using a live cd to get it up to 3.8TB...that was as far as I took it as I want to get the other problems fixed first... Unfortunately, due to my determination (aka: sore loser) one of two options exist...this WILL work...or one of us is going to die trying... Is it just me, or does everyone try to plead, reason, and insult their equipment...I swear to god, it derives pleasure from frustration... -Original Message- From: Benson Wong [mailto:[EMAIL PROTECTED] Sent: Thursday, April 14, 2005 6:29 PM To: [EMAIL PROTECTED] Cc: freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions So theoretically it should go over 1000TB.I've conducted several bastardized installations due to sysinstall not being able to do anything over the 2TB limit by creating the partition ahead of time.I am going to be attacking this tonight and my efforts will be primarily focused on creating one large 5.8TB slice..wish me luck!! PS: Muhaa haa haa! You're probably going to run into boo hoo hoo hoo. Most likely you won't be able to get over the 2TB limit. Also don't use sysinstall, I was never able to get it to work well. Probably because my arrays were mounted over fiber channel and fdisk craps out. This is what I did: dd if=/dev/zero of=/dev/da0 bs=1k count=1 disklabel -rw da0 audo newfs /dev/da0 That creates one large slice, UFS2, for FreeBSD. Let know if you get it over 2TB, I was never able to have any luck. Another reason you might want to avoid a super large file system is that UFS2 is not journaling. If the server crashes it will take fschk a LONG time to check all those inodes! Ben. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
Sorry for the delay in response.I had to go to the IRS today.OMFG.what a model of inefficiency. I am having some minor hardware issues with the build that we are going to be working on to get corrected first.but I will def keep everyone informed on what is going on.. _ From: Nick Pavlica [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 9:37 AM To: [EMAIL PROTECTED] Cc: Benson Wong; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions I am going to be attacking this tonight and my efforts will be primarily focused on creating one large 5.8TB slice..wish me luck!! How did this go? Were you able to create the very large slice? --Nick _ From: Nick Pavlica [mailto:[EMAIL PROTECTED] Sent: Thursday, April 14, 2005 2:49 PM To: Benson Wong Cc: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions Is there any limitations that would prevent a single volume that large? (if I remember there is a 2TB limit or something) 2TB is the largest for UFS2. 1TB is the largest for UFS1. Is the 2TB limit that you mention only for x86? This file system comparison lists the maximum size to be much larger (http://en.wikipedia.org/wiki/Comparison_of_file_systems ). --Nick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
Interesting... gpt add [-b number] [-i index] [-s count] [-t type] device ... The add command allows the user to add a new partition to an existing table. By default, it will create a UFS partition covering the first available block of an unused disk space. The command-specific options can be used to control this behaviour. I am assuming that the docs were not updated to reflect that its talking about UFS2? Or is it actually correct? -Original Message- From: Nick Evans [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 9:45 AM To: [EMAIL PROTECTED] Cc: 'Nick Pavlica'; 'Benson Wong'; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions On Thu, 14 Apr 2005 17:13:48 -0500 Edgar Martinez [EMAIL PROTECTED] wrote: Benson..GREAT RESPONSE!! I Don't think I could have done any better myself. Although I knew most of the information you provided, it was good to know that my knowledge was not very far off. It's also reassuring that I'm not the only nut job building ludicrous systems.. Nick, I believe that we may have some minor misinformation on our hands.. I refer you both to http://www.freebsd.org/projects/bigdisk/ which according to the page. When the UFS filesystem was introduced to BSD in 1982, its use of 32 bit offsets and counters to address the storage was considered to be ahead of its time. Since most fixed-disk storage devices use 512 byte sectors, 32 bits allowed for 2 Terabytes of storage. That was an almost un-imaginable quantity for the time. But now that 250 and 400 Gigabyte disks are available at consumer prices, it's trivial to build a hardware or software based storage array that can exceed 2TB for a few thousand dollars. The UFS2 filesystem was introduced in 2003 as a replacement to the original UFS and provides 64 bit counters and offsets. This allows for files and filesystems to grow to 2^73 bytes (2^64 * 512) in size and hopefully be sufficient for quite a long time. UFS2 largely solved the storage size limits imposed by the filesystem. Unfortunately, many tools and storage mechanisms still use or assume 32 bit values, often keeping FreeBSD limited to 2TB. So theoretically it should go over 1000TB.I've conducted several bastardized installations due to sysinstall not being able to do anything over the 2TB limit by creating the partition ahead of time.I am going to be attacking this tonight and my efforts will be primarily focused on creating one large 5.8TB slice..wish me luck!! PS: Muhaa haa haa! You'll need to use GPT to make this work for anything over 2TB. Man gpt Nick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
OK...so now we are going into some new territory...I am curious if you would care to elaborate a bit more...I am intrigued...if anyone wants me to do some experiments or test something, let me know...I for one welcome any attempts at pushing any limits or trying new things... -Original Message- From: Dan Nelson [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 11:18 AM To: Nick Evans Cc: [EMAIL PROTECTED]; 'Benson Wong'; 'Nick Pavlica'; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions In the last episode (Apr 15), Nick Evans said: You'll need to use GPT to make this work for anything over 2TB. Man gpt Or don't bother with a partition table at all, which makes growing the filesystem later on quite a bit easier. -- Dan Nelson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
I don't think I have ever done that...or even considered that was possible...The controller does support growing the array...Guess I'll give it a shot starting with 2TB and then grow it in increments to see how it behaves...any suggestions for newfs'in the device directly?? -Original Message- From: Dan Nelson [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 12:15 PM To: Edgar Martinez Cc: 'Nick Evans'; 'Benson Wong'; 'Nick Pavlica'; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions In the last episode (Apr 15), Edgar Martinez said: OK...so now we are going into some new territory...I am curious if you would care to elaborate a bit more...I am intrigued...if anyone wants me to do some experiments or test something, let me know...I for one welcome any attempts at pushing any limits or trying new things... If your array is just going to used for one large filesystem, you can skip any partitioning steps and newfs the base device directly. then if you decide to grow the array (and if your controller supports nondestructive resizing), you can use growfs to expand the filesystem without the extra step of manually adjusting a partition table. -- Dan Nelson [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
I was hoping that 5.4 would be out by the time I started this project.I'll give it a shot to see how it behaves. _ From: Nick Pavlica [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 12:15 PM To: [EMAIL PROTECTED] Cc: Dan Nelson; Nick Evans; Benson Wong; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions On 4/15/05, Edgar Martinez [EMAIL PROTECTED] wrote: OK...so now we are going into some new territory...I am curious if you would care to elaborate a bit more...I am intrigued...if anyone wants me to do some experiments or test something, let me know...I for one welcome any attempts at pushing any limits or trying new things... I would help do some testing but I don't have any storage that large at the moment. I curious how 5.4RC2 or handles very large volumes. Have you already tried fdisk, newfs ? --Nick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
5.8TB RAID5 SATA Array Questions
All, I have a project in which I have purchased the hardware to build a massive file server (specifically for video). The array from all estimates will come in at close to 5.8TB after overheard and formatting. Questions are: What Version of BSD (5.3, 5.4, 4.X)? What should the stripe size be for the array for speed when laying down video streams? What filesystem? Is there any limitations that would prevent a single volume that large? (if I remember there is a 2TB limit or something) The idea is to provide as much network storage as possible as fast as possible, any particular service? (SMB. NFS, ETC) Raid controller: 3Ware 9500S-12MI Thanks! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: 5.8TB RAID5 SATA Array Questions
Benson..GREAT RESPONSE!! I Don't think I could have done any better myself. Although I knew most of the information you provided, it was good to know that my knowledge was not very far off. It's also reassuring that I'm not the only nut job building ludicrous systems.. Nick, I believe that we may have some minor misinformation on our hands.. I refer you both to http://www.freebsd.org/projects/bigdisk/ which according to the page. When the UFS filesystem was introduced to BSD in 1982, its use of 32 bit offsets and counters to address the storage was considered to be ahead of its time. Since most fixed-disk storage devices use 512 byte sectors, 32 bits allowed for 2 Terabytes of storage. That was an almost un-imaginable quantity for the time. But now that 250 and 400 Gigabyte disks are available at consumer prices, it's trivial to build a hardware or software based storage array that can exceed 2TB for a few thousand dollars. The UFS2 filesystem was introduced in 2003 as a replacement to the original UFS and provides 64 bit counters and offsets. This allows for files and filesystems to grow to 2^73 bytes (2^64 * 512) in size and hopefully be sufficient for quite a long time. UFS2 largely solved the storage size limits imposed by the filesystem. Unfortunately, many tools and storage mechanisms still use or assume 32 bit values, often keeping FreeBSD limited to 2TB. So theoretically it should go over 1000TB.I've conducted several bastardized installations due to sysinstall not being able to do anything over the 2TB limit by creating the partition ahead of time.I am going to be attacking this tonight and my efforts will be primarily focused on creating one large 5.8TB slice..wish me luck!! PS: Muhaa haa haa! _ From: Nick Pavlica [mailto:[EMAIL PROTECTED] Sent: Thursday, April 14, 2005 2:49 PM To: Benson Wong Cc: [EMAIL PROTECTED]; freebsd-questions@freebsd.org Subject: Re: 5.8TB RAID5 SATA Array Questions Is there any limitations that would prevent a single volume that large? (if I remember there is a 2TB limit or something) 2TB is the largest for UFS2. 1TB is the largest for UFS1. Is the 2TB limit that you mention only for x86? This file system comparison lists the maximum size to be much larger (http://en.wikipedia.org/wiki/Comparison_of_file_systems). --Nick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
OK, removed all misc cards, devices, recabled...attempting to reinstall with 5.3, and I continue to get a kernel: priviledged instruction fault followed by a reboot...RAM is brand new Patriot 2-2-2-5...mem timings? -Original Message- From: Kris Kennaway [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 5:23 AM To: Edgar Martinez Cc: freebsd-questions@freebsd.org Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) On Wed, Apr 13, 2005 at 01:10:42AM -0500, Edgar Martinez wrote: All, I cant begin to tell you how horrible of a time I have had trying to get this system installed and running. Sysinstall kept throwing up a privilege fault kernel error randomly (7sed...4m.7m..etc), and after I go fast enough to get lucky to an install complete..the system then spends its time periodically rebooting.this is the first venture into AMD64 turf as I historically stick with i386.so any pointers.gotchas.tweaks or tips.. please let me know..I really don't want to give up, so I want to see what can be done to stabilize this. Sounds like the usual bad hardware story..check RAM, power supply, CPU cooling, cabling, etc. kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
Its using the default bios settings and nothing is overclocked at all.. -Original Message- From: Trevor Sullivan [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 9:00 AM To: [EMAIL PROTECTED] Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) Edgar Martinez wrote: OK, removed all misc cards, devices, recabled...attempting to reinstall with 5.3, and I continue to get a kernel: priviledged instruction fault followed by a reboot...RAM is brand new Patriot 2-2-2-5...mem timings? -Original Message- From: Kris Kennaway [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 5:23 AM To: Edgar Martinez Cc: freebsd-questions@freebsd.org Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) On Wed, Apr 13, 2005 at 01:10:42AM -0500, Edgar Martinez wrote: All, I cant begin to tell you how horrible of a time I have had trying to get this system installed and running. Sysinstall kept throwing up a privilege fault kernel error randomly (7sed...4m.7m..etc), and after I go fast enough to get lucky to an install complete..the system then spends its time periodically rebooting.this is the first venture into AMD64 turf as I historically stick with i386.so any pointers.gotchas.tweaks or tips.. please let me know..I really don't want to give up, so I want to see what can be done to stabilize this. Sounds like the usual bad hardware story..check RAM, power supply, CPU cooling, cabling, etc. kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Is this computer overclocked at all? I would highly recommend running it at the speeds it was meant to...you can avoid a lot of errors that way. -Trevor ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
New error before death Panic: page fault -Original Message- From: Trevor Sullivan [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 9:00 AM To: [EMAIL PROTECTED] Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) Edgar Martinez wrote: OK, removed all misc cards, devices, recabled...attempting to reinstall with 5.3, and I continue to get a kernel: priviledged instruction fault followed by a reboot...RAM is brand new Patriot 2-2-2-5...mem timings? -Original Message- From: Kris Kennaway [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 5:23 AM To: Edgar Martinez Cc: freebsd-questions@freebsd.org Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) On Wed, Apr 13, 2005 at 01:10:42AM -0500, Edgar Martinez wrote: All, I cant begin to tell you how horrible of a time I have had trying to get this system installed and running. Sysinstall kept throwing up a privilege fault kernel error randomly (7sed...4m.7m..etc), and after I go fast enough to get lucky to an install complete..the system then spends its time periodically rebooting.this is the first venture into AMD64 turf as I historically stick with i386.so any pointers.gotchas.tweaks or tips.. please let me know..I really don't want to give up, so I want to see what can be done to stabilize this. Sounds like the usual bad hardware story..check RAM, power supply, CPU cooling, cabling, etc. kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Is this computer overclocked at all? I would highly recommend running it at the speeds it was meant to...you can avoid a lot of errors that way. -Trevor ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
Yep, Tried 5.3 mini, two different 5.3 Disc 1, 5.4RC2 Disc 1 Strangely enough, I disabled in the bios the CPU Cache and although the system is sluggish and slow, it has not freaked out yet.. -Original Message- From: Trevor Sullivan [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 9:13 AM To: [EMAIL PROTECTED] Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) Edgar Martinez wrote: New error before death Panic: page fault -Original Message- From: Trevor Sullivan [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 9:00 AM To: [EMAIL PROTECTED] Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) Edgar Martinez wrote: OK, removed all misc cards, devices, recabled...attempting to reinstall with 5.3, and I continue to get a kernel: priviledged instruction fault followed by a reboot...RAM is brand new Patriot 2-2-2-5...mem timings? -Original Message- From: Kris Kennaway [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 5:23 AM To: Edgar Martinez Cc: freebsd-questions@freebsd.org Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) On Wed, Apr 13, 2005 at 01:10:42AM -0500, Edgar Martinez wrote: All, I cant begin to tell you how horrible of a time I have had trying to get this system installed and running. Sysinstall kept throwing up a privilege fault kernel error randomly (7sed...4m.7m..etc), and after I go fast enough to get lucky to an install complete..the system then spends its time periodically rebooting.this is the first venture into AMD64 turf as I historically stick with i386.so any pointers.gotchas.tweaks or tips.. please let me know..I really don't want to give up, so I want to see what can be done to stabilize this. Sounds like the usual bad hardware story..check RAM, power supply, CPU cooling, cabling, etc. kris ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Is this computer overclocked at all? I would highly recommend running it at the speeds it was meant to...you can avoid a lot of errors that way. -Trevor Is there perhaps a problem with the burned CD? Did you check the hash after downloading it, and verify the cd's contents after burning it? Sorry it's a stab in the dark, but maybe it'll give you a push in the direction of making it work =D -Trevor ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
I appreciate your help! There is NO stupid question or answer...as far as I am concerned ANY help is always WELCOME help...THANKS So, yeah being from the i386 world, I am also aware of the timing quirks in the intel world. The CPU is a 754 pin...specs below..hope this sheds some light on the situation...if anyone wants me to throw up some debug info let me know!! AMD Athlon 64 3000+, 1MB L2 Cache, 64-bit Processor for DTR Notebooks - OEM Model# AMA3000BEX5AP Item # N82E16819103444 Specifications: Model: AMD Athlon 64 3000+ Core: ClawHammer Operating Frequency: 1.8 GHz FSB: Integrated into Chip Cache: L1/64K+64K; L2/1MB Voltage: 1.5V Process: 0.13Micron Socket: Socket 754 Multimedia Instruction: MMX, SSE, SSE2, 3DNOW!, 3DNOW!+ Packaging: OEM(Processor Only) PATRIOT Extreme Performance 184-Pin 512MB DDR PC-3200 w/ XBL Technology, Model PEP5123200+XBL - Retail Model# PEP5123200+XBL Item # N82E16820220036 Specifications: Manufacturer: PDP Systems Speed: DDR400(PC3200) Type: 184-Pin DDR SDRAM Error Checking: Non-ECC Registered/Unbuffered: Unbuffered Cas Latency: 2-2-2-5 T1 Support Voltage: 2.8V Bandwidth: 3.2GB/s Organization: 64M x 64 -Bit Warranty: Lifetime -Original Message- From: Alex Zbyslaw [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 9:40 AM To: [EMAIL PROTECTED] Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) Edgar Martinez wrote: OK, removed all misc cards, devices, recabled...attempting to reinstall with 5.3, and I continue to get a kernel: priviledged instruction fault followed by a reboot...RAM is brand new Patriot 2-2-2-5...mem timings? I don't know this brand of RAM, and I may be completely wrong but... When looking at 939 pin AMD CPUs, and what memory to get, I noticed that for one major quality RAM maker, the timings they recommended for Intel CPUs were faster than those they recommended for AMD CPUs and your figures look eerily familiar. It was something like 2-2-2-5 for Intel and 2-2.5-2-5 for AMD. I'm no expert on RAM, but your error does sound very hardware related. Can you check the RAM manufacturer's web site to see if they say anything? Also, some motherboards can be very picky about RAM. Is it worth trying with just one RAM chip? Does the machine POST ok with full memory tests on? (It doesn't prove anything, but if a chip is really duff it should find it). A little heretical, I realise, but is there the possibility of trying another OS (Windows, Linux) just to see if you have the same kinds of problems? I'm not copying to the list because I'm no real expert on this kind of thing; just stuff picked up from background reading. If you try any of this and it works can you post back to the list what finally worked? Best, --Alex ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
Yep first thing I assumed...ACPI was disabled...both in BIOS and via MENU...no joy...UDMA disabled...in fact.. PATA DISABLED (after install via CD) USB DISABLED FDD DIABLED APM DIABLED SMART DISABLED LAN DISABLED SOUND DISABLED FIREWIRE DISABLED =) If you cant tell, I have literally installed over a hundred FBSD boxes and have encountered TONS of caveats...however this one has def got me stumped..after the install, the system seems to be holding stable with the BIOS INTERNAL/EXTERNAL cache disabled...its SLLLOOOWWW but stable... -Original Message- From: NMH [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 2:19 PM To: [EMAIL PROTECTED]; questions Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) --- Edgar Martinez [EMAIL PROTECTED] wrote: All, I cant begin to tell you how horrible of a time I have had trying to get this system installed and running. Sysinstall kept throwing up a privilege fault kernel error randomly (7sed...4m.7m..etc), and after I go fast enough to get lucky to an install complete..the system then spends its time periodically rebooting.this is the first venture into AMD64 turf as I historically stick with i386.so any pointers.gotchas.tweaks or tips.. please let me know..I really don't want to give up, so I want to see what can be done to stabilize this. Unless someone else can vouch for that MB, it can be a suspect as well. While AMD is good, not all MB's for them are. Also ACPI can cause weird stuff like that too. Perhaps try turning that off or try other settings in the BIOS. I had that problem once. It actually did then when ACPI was turned off in the bios. Best of luck. NMH MSI K8T Neo AMD64 3000 w/1MB Thanks! ## SNIP ## The Large Print Giveth And The Small Print Taketh Away -- Anon __ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/ ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install)
I think I have nailed it...somewhat... So I set it up so I could ssh to it from my office and try to mess with it...ran solid as a rock...I think got home tonight and checked my logs...nothing bad...so I THEN rebooted went into BIOS and enabled the cache...BAM...errors out every time...threw in ubuntu...craptastic...DISABLED the CACHE...everything smooth...sooo the question now is...MB or CPU?? The CPU is listed as DTR...OMFG WTF is DTR?? (acro-cursing intended..) Model: AMD Athlon 64 3000+ DTR Core: ClawHammer Operating Frequency: 1.8 GHz FSB: Integrated into Chip Cache: L1/64K+64K; L2/1MB Voltage: 1.5V Process: 0.13Micron Socket: Socket 754 Multimedia Instruction: MMX, SSE, SSE2, 3DNOW!, 3DNOW!+ Packaging: OEM(Processor Only) -Original Message- From: jason henson [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 13, 2005 7:20 PM To: [EMAIL PROTECTED] Cc: freebsd-questions@freebsd.org Subject: Re: AMD64 (former i386 convert)+ FreeBSD various issues in 5.3, 5.4 (pre+post install) Edgar Martinez wrote: All, I cant begin to tell you how horrible of a time I have had trying to get this system installed and running. Sysinstall kept throwing up a privilege fault kernel error randomly (7sed...4m.7m..etc), and after I go fast enough to get lucky to an install complete..the system then spends its time periodically rebooting.this is the first venture into AMD64 turf as I historically stick with i386.so any pointers.gotchas.tweaks or tips.. please let me know..I really don't want to give up, so I want to see what can be done to stabilize this. MSI K8T Neo AMD64 3000 w/1MB I was just reading the archives this week at freebsd.org and it leads me to believe msi make crap boards. They can not handle tough loads or lots of ram. I think it was in the amd64 list under a heading that mentioned 8gig of ram. There were several developers that just trashed there msi boards and all the rest of there hardware worked fine in a new board. You should look it up. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Promise TX2 Rebuild via atacontrol??
I apologize...I usually am the one complaining to others about versioning...my bad... I do not think that the drive is at fault as much as the controller. I have had to deploy well over one hundred of the exact same configuration, and historically, it's always been the controller that just drops the drive for some reason. I usually end up re-mirroring the array on the old dead drive and the system will continue on its merry way (last one is over 1 year old since last resync)...hence you see why I am curious if I could just use atacontrol instead of taking the unit offline and spending 45min waiting for the resync TECHNOJARGON BELOW FBSD 5.3 STABLE Promise TX2 Orig BIOS ad4: 114473MB WDC WD1200JB-00DUA3/75.13B75 [232581/16/63] at ata2-master UDMA100 ad6: 114473MB WDC WD1200JB-00DUA3/75.13B75 [232581/16/63] at ata3-master UDMA100 ar0: 114440MB ATA RAID1 array [14589/255/63] status: READY subdisks: disk0 READY on ad4 at ata2-master disk1 READY on ad6 at ata3-master Mounting root from ufs:/dev/ar0s1a ad4: TIMEOUT - WRITE_DMA retrying (2 retries left) LBA=5861119 ad4: FAILURE - WRITE_DMA timed out ar0: WARNING - mirror lost -Original Message- From: Emanuel Strobl [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 05, 2005 4:15 AM To: freebsd-questions@freebsd.org; [EMAIL PROTECTED] Subject: Re: Promise TX2 Rebuild via atacontrol?? Am Montag, 4. April 2005 23:29 schrieb Edgar Martinez: All, I have a failed member in a RAID1 array and using atacontrol can see that the status is degraded. I am curious if I can use atacontrol to rebuild the array if the original array was built using the Promise BIOS utility. If I tell atacontrol to rebuild.will it corrupt my data or catch fire and explode?? You don't tell us what version you use, but promise is supported very well, even in atamkII in 6-current. If you replace the failed drive it sould be automatically rebuilt, the `atacontrol rebuild ar0` doesn't work as long as you (in 5.x) used addspare or the controller found a good spare drive. In 4.x you don't have the addspare option, you have to `atacontrol detach 3` and reattach it the same way to get recognized and inserted as spare in an existing array. -Harry Cheers! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Promise TX2 Rebuild via atacontrol??
All, I have a failed member in a RAID1 array and using atacontrol can see that the status is degraded. I am curious if I can use atacontrol to rebuild the array if the original array was built using the Promise BIOS utility. If I tell atacontrol to rebuild.will it corrupt my data or catch fire and explode?? Cheers! ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
