Re: problem on 1TB filesystem RAID 5 3ware
On 3/22/2003 at 2:17 PM Bruce Campbell wrote: |Some more test results: | |22 Mar 2003 - test with RAID 5 with 4 * WD 200GB with Write Cache disabled |*succeeded*. Write Cache can be disabled through the 3ware BIOS, or the |3ware |web management tool. Raw write performance to the array dropped from |3KBytes/Second to 4500KBytes/Second, however this did not impact the |test |significantly, as the test involved copying data via an NFS mount on a 100 |MBit/second network. The effective speed of the NFS copy dropped from |around |5000 KBytes/Second to about 4500 KBytes/Second with Write Cache disabled. | |Thread here: | |http://oss.sgi.com/projects/xfs/mail_archive/200211/msg00056.html | |suggests firmware/driver mismatches can cause trouble, and someone |else who had trouble found turning off write cache fixed it. | | |All my info on this problem being kept here: | |http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerProblem = Thanks for your follow-up, but I'd caution you regarding the serendipitous it's working if I do this aspect of your discovery. Those fixes have a nasty habit of disappearing when something else changes in the system I have switched over to RHLinux for the server using the 3Ware card, which is ironic because I left RHLinux when I switched to FreeBSD a year or so ago. :-) I'd much prefer to run my RAID under FreeBSD, but I prefer to use supported hardware even more. The 3Ware card is a high-end, high-performance IDE RAID card; as such, I am rather surprised that it is not properly supported under FreeBSD. n To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Some more test results: 22 Mar 2003 - test with RAID 5 with 4 * WD 200GB with Write Cache disabled *succeeded*. Write Cache can be disabled through the 3ware BIOS, or the 3ware web management tool. Raw write performance to the array dropped from 3KBytes/Second to 4500KBytes/Second, however this did not impact the test significantly, as the test involved copying data via an NFS mount on a 100 MBit/second network. The effective speed of the NFS copy dropped from around 5000 KBytes/Second to about 4500 KBytes/Second with Write Cache disabled. Thread here: http://oss.sgi.com/projects/xfs/mail_archive/200211/msg00056.html suggests firmware/driver mismatches can cause trouble, and someone else who had trouble found turning off write cache fixed it. All my info on this problem being kept here: http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerProblem Quoting MikeM [EMAIL PROTECTED]: Jim King [EMAIL PROTECTED] wrote: Which tells me that all of the work in the last year has been maintenance related to changes within FreeBSD itself, and not any updates for 3Ware functionality, e.g no support for firmware 7.5.x on the 7000 series controllers, and no support for the 8000 series controllers. If the above is true, perhaps the hardware guide should be modified. It currently says that the 3Ware 7000 series is supported. I, for one, purchased a 3Ware controller for my FreeBSD server based upon the misleading hardware guide. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hardware in the body of the message -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
I've been using the 3Ware cards for 2 years on FreeBSD... We had a drive go south a few weeks ago, and the 3dmd (3Ware daemon) reported the problem to me. I had a local hardware tech go out and replace the drive, boot up the system, and then over the net, I rebuilt the array. It was back up and running after about 20 minutes. I did contact 3Ware tech support when this first happened, to make sure that I was following the steps of rebuilding the array correctly. They responded within 24 hours, and told me yes, I was doing this correctly. And I did mention to them it was FreeBSD. At no time, did they ever mention that they didn't support FreeBSD. And that 3Ware daemon works great... Have it running on all of my systems all over the country. Peter At 08:22 PM 3/20/2003 -0600, you wrote: Bruce Campbell wrote: I openned a case with 3ware tech support and they responded: We do not support FreeBSD plus the current driver for FreeBSD has not been updated for some time to keep up with firmware changes. Please try linux instead. So I guess I will try that and see what happens. I got NO response from 3Ware when I asked them about a newer version of their management daemon for FreeBSD. I consider the 3Ware cards to be essentially unsupported on FreeBSD. Jim To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message -- Peter Elsner [EMAIL PROTECTED] Vice President Of Customer Service (And System Administrator) 1835 S. Carrier Parkway Grand Prairie, Texas 75051 (972) 263-2080 - Voice (972) 263-2082 - Fax (972) 489-4838 - Cell Phone (425) 988-8061 - eFax I worry about my child and the Internet all the time, even though she's too young to have logged on yet. Here's what I worry about. I worry that 10 or 15 years from now, she will come to me and say Daddy, where were you when they took freedom of the press away from the Internet? -- Mike Godwin Unix IS user friendly... It's just selective about who its friends are. System Administration - It's a dirty job, but somebody said I had to do it. If you receive something that says 'Send this to everyone you know, pretend you don't know me. Standard $500/message proofreading fee applies for UCE. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
RE: problem on 1TB filesystem RAID 5 3ware
Have a look at: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/twe/ Lawrence Farr EPC Direct Limited -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bruce Campbell Sent: 21 March 2003 14:08 To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: problem on 1TB filesystem RAID 5 3ware The author of the twe driver, Michael Smith, confirms he hasn't worked on the driver for several years now. Quoting Bruce Campbell [EMAIL PROTECTED]: I openned a case with 3ware tech support and they responded: We do not support FreeBSD plus the current driver for FreeBSD has not been updated for some time to keep up with firmware changes. Please try linux instead. So I guess I will try that and see what happens. Quoting Simon [EMAIL PROTECTED]: I have a hard time believing that hardware implementation of RAID5 would corrupt files over RAID10, perhaps your 3ware card/its firmware is malfunctioning, but anything is possible. Well, I'll have to see for myself, I'm about to build RAID5 NAS using maxtor drives and 7500-8 3ware card. -Simon On Tue, 18 Mar 2003 08:05:36 -0500, Bruce Campbell wrote: Tested with RAID 10 instead of RAID 5, success ! RAID 10 arrays tested: 6xWD200GB and 8xWD200GB both worked RAID 5 arrays tested: 6xWD200GB and 4xWD200GB both failed Note: 3ware lists the WD 200GB disk as Under Test. (ie they have not yet given it a Compatible rating) details of tests and the procedure to detect the failure etc at http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/Backup ServerProblem I still have to try an officially approved drive. This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hardware in the body of the message -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hardware in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
But the version of 3dmd for FreeBSD is not a current version and lacks some features found in 3dmd for Linux and Windows. My situation was that I needed one of the new features in order to rebuild a RAID1 mirror (ignore errors when rebuilding; my good drive had a bad sector out in unused space). I won't be installing anymore FreeBSD/3Ware setups. Jim Peter Elsner wrote: I've been using the 3Ware cards for 2 years on FreeBSD... We had a drive go south a few weeks ago, and the 3dmd (3Ware daemon) reported the problem to me. I had a local hardware tech go out and replace the drive, boot up the system, and then over the net, I rebuilt the array. It was back up and running after about 20 minutes. I did contact 3Ware tech support when this first happened, to make sure that I was following the steps of rebuilding the array correctly. They responded within 24 hours, and told me yes, I was doing this correctly. And I did mention to them it was FreeBSD. At no time, did they ever mention that they didn't support FreeBSD. And that 3Ware daemon works great... Have it running on all of my systems all over the country. Peter At 08:22 PM 3/20/2003 -0600, you wrote: Bruce Campbell wrote: I openned a case with 3ware tech support and they responded: We do not support FreeBSD plus the current driver for FreeBSD has not been updated for some time to keep up with firmware changes. Please try linux instead. So I guess I will try that and see what happens. I got NO response from 3Ware when I asked them about a newer version of their management daemon for FreeBSD. I consider the 3Ware cards to be essentially unsupported on FreeBSD. Jim To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Which tells me that all of the work in the last year has been maintenance related to changes within FreeBSD itself, and not any updates for 3Ware functionality, e.g no support for firmware 7.5.x on the 7000 series controllers, and no support for the 8000 series controllers. Jim Lawrence Farr wrote: Have a look at: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/twe/ Lawrence Farr EPC Direct Limited -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bruce Campbell Sent: 21 March 2003 14:08 To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: problem on 1TB filesystem RAID 5 3ware The author of the twe driver, Michael Smith, confirms he hasn't worked on the driver for several years now. Quoting Bruce Campbell [EMAIL PROTECTED]: I openned a case with 3ware tech support and they responded: We do not support FreeBSD plus the current driver for FreeBSD has not been updated for some time to keep up with firmware changes. Please try linux instead. So I guess I will try that and see what happens. Quoting Simon [EMAIL PROTECTED]: I have a hard time believing that hardware implementation of RAID5 would corrupt files over RAID10, perhaps your 3ware card/its firmware is malfunctioning, but anything is possible. Well, I'll have to see for myself, I'm about to build RAID5 NAS using maxtor drives and 7500-8 3ware card. -Simon On Tue, 18 Mar 2003 08:05:36 -0500, Bruce Campbell wrote: Tested with RAID 10 instead of RAID 5, success ! RAID 10 arrays tested: 6xWD200GB and 8xWD200GB both worked RAID 5 arrays tested: 6xWD200GB and 4xWD200GB both failed Note: 3ware lists the WD 200GB disk as Under Test. (ie they have not yet given it a Compatible rating) details of tests and the procedure to detect the failure etc at http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/Backup ServerProblem I still have to try an officially approved drive. This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hardware in the body of the message -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Jim King [EMAIL PROTECTED] wrote: Which tells me that all of the work in the last year has been maintenance related to changes within FreeBSD itself, and not any updates for 3Ware functionality, e.g no support for firmware 7.5.x on the 7000 series controllers, and no support for the 8000 series controllers. If the above is true, perhaps the hardware guide should be modified. It currently says that the 3Ware 7000 series is supported. I, for one, purchased a 3Ware controller for my FreeBSD server based upon the misleading hardware guide. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
I openned a case with 3ware tech support and they responded: We do not support FreeBSD plus the current driver for FreeBSD has not been updated for some time to keep up with firmware changes. Please try linux instead. So I guess I will try that and see what happens. Quoting Simon [EMAIL PROTECTED]: I have a hard time believing that hardware implementation of RAID5 would corrupt files over RAID10, perhaps your 3ware card/its firmware is malfunctioning, but anything is possible. Well, I'll have to see for myself, I'm about to build RAID5 NAS using maxtor drives and 7500-8 3ware card. -Simon On Tue, 18 Mar 2003 08:05:36 -0500, Bruce Campbell wrote: Tested with RAID 10 instead of RAID 5, success ! RAID 10 arrays tested: 6xWD200GB and 8xWD200GB both worked RAID 5 arrays tested: 6xWD200GB and 4xWD200GB both failed Note: 3ware lists the WD 200GB disk as Under Test. (ie they have not yet given it a Compatible rating) details of tests and the procedure to detect the failure etc at http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerProblem I still have to try an officially approved drive. This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hardware in the body of the message -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Bruce Campbell wrote: I openned a case with 3ware tech support and they responded: We do not support FreeBSD plus the current driver for FreeBSD has not been updated for some time to keep up with firmware changes. Please try linux instead. So I guess I will try that and see what happens. I got NO response from 3Ware when I asked them about a newer version of their management daemon for FreeBSD. I consider the 3Ware cards to be essentially unsupported on FreeBSD. Jim To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Tested with RAID 10 instead of RAID 5, success ! RAID 10 arrays tested: 6xWD200GB and 8xWD200GB both worked RAID 5 arrays tested: 6xWD200GB and 4xWD200GB both failed Note: 3ware lists the WD 200GB disk as Under Test. (ie they have not yet given it a Compatible rating) details of tests and the procedure to detect the failure etc at http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerProblem I still have to try an officially approved drive. This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Not solved this yet, but I have determined a few things that the problem isn't. Info at: http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerProblem Tested with soft updates off and on, fails in either case, so that isn't it. Seems like the problem is either: - 3ware card or driver - something to do with the large filesystem Quoting Bruce Campbell [EMAIL PROTECTED]: File corruption on 2 identical systems, designed to be backup servers to contain dumps of other systems: FreeBSD ecserv18.uwaterloo.ca 4.7-RELEASE FreeBSD 4.7-RELEASE #0: Wed Oct 9 15:08:34 GMT 2002 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC i386 with 1TB /backup partition, on a 3ware 7500-8 ATA RAID card, RAID 5: Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/twed0s1a20644846906552 18086708 5%/ procfs 4 4 0 100%/proc /dev/twed0s1e 938819776 279031856 58468233832%/backup disks are 6 x Western Digital 2000JB (200GB) I ran tests on /backup for 10 days on each system (fill disk with 50GB files of pseudo random data, then reading them all back and verify contents, then erase, then start over). Tests ran perfectly. details on hardware config at: http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerHardware Then, I was ready to put the systems into production, so I copied data from my 2 older backup servers (which have 360GB vinum partitions) and after copying the data (approx 250GB in 325 files) about a dozen files were corrupt after the copy. I copied via an NFS mount. All corruption started on a 64K boundary, except one which was on a 16K boundary. Recopied the dozen corrupt files, and then only 6 were corrupt. Same problem on both systems, each which copied from a different source server. File seems corrupt to the end after first corruption starts, I have not looked for a pattern to see if it is another files contents, or misplaced contents from the same file. fsck shows no problems Restarted my test filling with 50GB files again, has run perfectly. I plan to try: - turn off soft updates - RAID 10 instead of 5 - different file system parameters, for example I don't need 100 million inodes. - rcp'ing the files - staring at computer screen By the way, 3ware has not officially approved the WD 200GB drive last time I checked. Lots of good experience with the motherboard (ASUS P4S533) and network card (Intel Pro/100). Lots of good experience with vinum striped partitions of smaller size (360GB) Does anyone have any suggestions ? -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
Re: problem on 1TB filesystem RAID 5 3ware
Quoting Simon [EMAIL PROTECTED]: I can only hope I don't have the same issue. I'm currently building a 1.75TB NAS to do daily backups using 3ware 7500-8 and maxtor drives. Tiny bit more info: - NFS was starting to be implicated, but on one of my backup servers I had let it run 2 dumps of our Network Appliance, basically: rsh netapp dump ... | gzip file and I tried gunzip -t to test the file, and both were corrupt. My backup system I've been running with vinum for a long time does a weekly gunzip -t on all files, and I've not seen a problem before. This also removes the network card from suspicion, as if it was the problem, the .gz file would still be valid (it would just be compressed garbage, but it would not be corrupt itself) Here is the program I wrote to test the partitions: http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BurnInProcedure (obviously not an outstanding test, since it passed my system) -Simon On Wed, 12 Mar 2003 20:38:13 -0500, Bruce Campbell wrote: File corruption on 2 identical systems, designed to be backup servers to contain dumps of other systems: FreeBSD ecserv18.uwaterloo.ca 4.7-RELEASE FreeBSD 4.7-RELEASE #0: Wed Oct 9 15:08:34 GMT 2002 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC i386 with 1TB /backup partition, on a 3ware 7500-8 ATA RAID card, RAID 5: Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/twed0s1a20644846906552 18086708 5%/ procfs 4 4 0 100%/proc /dev/twed0s1e 938819776 279031856 58468233832%/backup disks are 6 x Western Digital 2000JB (200GB) I ran tests on /backup for 10 days on each system (fill disk with 50GB files of pseudo random data, then reading them all back and verify contents, then erase, then start over). Tests ran perfectly. details on hardware config at: http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/BackupServerHardware Then, I was ready to put the systems into production, so I copied data from my 2 older backup servers (which have 360GB vinum partitions) and after copying the data (approx 250GB in 325 files) about a dozen files were corrupt after the copy. I copied via an NFS mount. All corruption started on a 64K boundary, except one which was on a 16K boundary. Recopied the dozen corrupt files, and then only 6 were corrupt. Same problem on both systems, each which copied from a different source server. File seems corrupt to the end after first corruption starts, I have not looked for a pattern to see if it is another files contents, or misplaced contents from the same file. fsck shows no problems Restarted my test filling with 50GB files again, has run perfectly. I plan to try: - turn off soft updates - RAID 10 instead of 5 - different file system parameters, for example I don't need 100 million inodes. - rcp'ing the files - staring at computer screen By the way, 3ware has not officially approved the WD 200GB drive last time I checked. Lots of good experience with the motherboard (ASUS P4S533) and network card (Intel Pro/100). Lots of good experience with vinum striped partitions of smaller size (360GB) Does anyone have any suggestions ? -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hardware in the body of the message -- Bruce Campbell Engineering Computing CPH-2374B University of Waterloo (519)888-4567 ext 5889 This mail sent through www.mywaterloo.ca To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-questions in the body of the message
