Re: A little story of failed raid5 (3ware 8000 series)
On Sun, Aug 26, 2007 at 11:38:19AM -0700, Darren Pilgrim wrote: Tom Judge wrote: Tom Samplonius wrote: The real solution is RAID scrubbing: a low level background process that reads every sector of every disk. All of the real RAID systems do this (usually scheduled weekly, or every other week). Most 3ware RAID card don't have this feature. So rather than not using RAID5 or RAID6 again, you should just not use 3ware anymore. If you use the 3dm2 management interface you can schedule verify and rebuild tasks to run on a regular basis. I think that 7500 series controllers can do this, 9500 and 9550's definitely can. Actually it's all 7/8/9xxx series cards. The 9xxx series cards also do auto-verify so there's no need to schedule the task. Sorry, I do not believe that is the case. I have a 8506-4LP and if I click on Management - Scheduling in the 3dm web interface it says (0x0C:0x0017): Scheduling is not supported on this controller model However you might be able to cheat using the tw_cli port to run the schedules out of cron rather than native on the card ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
3ware cards [Was: Re: A little story of failed raid5 (3ware 8000 series)]
Gary Palmer wrote: Darren Pilgrim wrote: Tom Judge wrote: If you use the 3dm2 management interface you can schedule verify and rebuild tasks to run on a regular basis. I think that 7500 series controllers can do this, 9500 and 9550's definitely can. Actually it's all 7/8/9xxx series cards. The 9xxx series cards also do auto-verify so there's no need to schedule the task. Sorry, I do not believe that is the case. I have a 8506-4LP and if I click on Management - Scheduling in the 3dm web interface it says (0x0C:0x0017): Scheduling is not supported on this controller model However you might be able to cheat using the tw_cli port to run the schedules out of cron rather than native on the card I was talking about the verify task, though I now realize that wasn't clear. Some of the older cards do only support tw_sched. The ability to perform data verification and media scans has been present since the 7xxx series. -- Darren Pilgrim ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Tom Judge wrote: Tom Samplonius wrote: The real solution is RAID scrubbing: a low level background process that reads every sector of every disk. All of the real RAID systems do this (usually scheduled weekly, or every other week). Most 3ware RAID card don't have this feature. So rather than not using RAID5 or RAID6 again, you should just not use 3ware anymore. If you use the 3dm2 management interface you can schedule verify and rebuild tasks to run on a regular basis. I think that 7500 series controllers can do this, 9500 and 9550's definitely can. Actually it's all 7/8/9xxx series cards. The 9xxx series cards also do auto-verify so there's no need to schedule the task. -- Darren Pilgrim ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
- Artem Kuchin [EMAIL PROTECTED] wrote: ... But i don't understand how and why it happened. ONly 6 hours ago (a night before) all those files were backed up fine w/o any read error. And now, right after replacing the driver and starting rebuild it said that there are bad sectors all over those file. How come? What happened to you was an extremely common occurrence. You had a disk develop a media failure sometime ago, but the controller never detected it, because that particular bad area was not read. Your backups worked because they never touched this portion of the disk (ex. empty space, meta data, etc). And then another drive developed a electronics failure, which is instantly detected, putting the array into a degraded mode. When you did a rebuild onto a replace drive, the controller discovered that there was a second failed disk, and this is unrecoverable. RAID, of any level, isn't magic. It is important to understand how it works, an realize that drives can passive fail. BTW, if you were using RAID1 or RAID10, you would likely have had the same problem (well, RAID10 can survive _some_ double-disk failures). RAID6 is the only RAID level that can survive failure of any two disks. The real solution is RAID scrubbing: a low level background process that reads every sector of every disk. All of the real RAID systems do this (usually scheduled weekly, or every other week). Most 3ware RAID card don't have this feature. So rather than not using RAID5 or RAID6 again, you should just not use 3ware anymore. Tom ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
- Martin Nilsson [EMAIL PROTECTED] wrote: That is what patrol read is intended to detect before it is a problem. In a RAID5 array the checksums are only used when reconstructing data, if you have a bad block in a checksum sector it will not be detected until a drive have failed and you try to rebuild the array, unfortunately at that time it is too late... Beware that OS software solutions like diskcheckd will not find this as it only reads the data, not the checksums, it must be done on the controller. This isn't really accurate. First of all, if the RAID controller isn't confirming checksums before giving the data to the OS, what is the checksum for exactly? It is supposed to be for detecting data corruption, so if the card isn't using the checksum, its kinda of useless. I know some RAID systems do fake their checksums, as they don't actually validate data against the checksums during normal reads because they don't have the processing power. I'd stay away from these type of systems (cough ... Blue Arc ... cough). Second, most RAID systems don't use their own checksums anymore. Netapp is quite famous for their ZCS (zone checksum) drives, and still uses a variation of this technology on their newer systems (which are using 512 sectors). But most RAID vendors just rely on the drives own error detection and correction systems (hamming code based usually, which is actually pretty solid). I'm pretty sure that that 3ware doesn't use any checksums. However, in this particular case, validating checksums would have been unhelpful, since the disk was unreadable. diskcheckd would have detected this issue. It would probably have prevented the problem, if it had been running previously. ZFS is also a good option. It has file level checksumming. ZFS never trusts the disks, and is super paranoid. And ZFS can do background scrubbing too. I can't wait for ZFS in FreeBSD 7, because ZFS in software is going to 10 x better than anything 3ware has. Regards, Martin Tom ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: A little story of failed raid5 (3ware 8000 series)
This isn't really accurate. First of all, if the RAID controller isn't confirming checksums before giving the data to the OS, what is the checksum for exactly? The checksum is used to recover the data in the event one piece of the data is lost. With all of the data but one piece, and the checksum, the data can be recovered. Confirming the checksum on every read would be a waste of time since the individual drives already checks the data for errors. It is supposed to be for detecting data corruption, so if the card isn't using the checksum, its kinda of useless. You are confused. Checking for data corruption is done, by checking if the *DATA* is corrupt. This does not require looking at the RAID5 checksum since the data has its own data checksum. I know some RAID systems do fake their checksums, as they don't actually validate data against the checksums during normal reads because they don't have the processing power. I'd stay away from these type of systems (cough ... Blue Arc ... cough). It has nothing to do with processing power. It's simply a waste. The RAID 5 checksum isn't for verifying the data, it's for recovering the data if it can't be read. Second, most RAID systems don't use their own checksums anymore. Netapp is quite famous for their ZCS (zone checksum) drives, and still uses a variation of this technology on their newer systems (which are using 512 sectors). But most RAID vendors just rely on the drives own error detection and correction systems (hamming code based usually, which is actually pretty solid). I'm pretty sure that that 3ware doesn't use any checksums. You are seriously confused. You are confusing the RAID 5 checksum with the drive data checksum. We are talking about making sure the RAID 5 checksums are readable so that, if a drive fails, the data can be reconstructed from the checksum. However, in this particular case, validating checksums would have been unhelpful, since the disk was unreadable. diskcheckd would have detected this issue. It would probably have prevented the problem, if it had been running previously. No, it would have saved him. The problem was he lost a drive, and checksums *ON* *OTHER* *DRIVES* were unreadable. Quite possibly they had been unreabable for months, but were never checked, since they are only *needed* to reconstruct the data. ZFS is also a good option. It has file level checksumming. ZFS never trusts the disks, and is super paranoid. And ZFS can do background scrubbing too. I can't wait for ZFS in FreeBSD 7, because ZFS in software is going to 10 x better than anything 3ware has. That wouuld not have helped him one bit. When the drive failed, the RAID 5 checksums on the other drives still would not have been scrubbed. The RAID 5 checksum (technically an XOR) is only needed to recover the RAID 5 array if a drive (or sector) fails. The only thing that will fix this is specifically verifying the RAID 5 checksum blocks. If a controller provides no way to do this, it is badly broken. If a verify operation does not verify the checksum blocks, it is broken. DS ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Tom Samplonius wrote: - Artem Kuchin [EMAIL PROTECTED] wrote: ... But i don't understand how and why it happened. ONly 6 hours ago (a night before) all those files were backed up fine w/o any read error. And now, right after replacing the driver and starting rebuild it said that there are bad sectors all over those file. How come? What happened to you was an extremely common occurrence. You had a disk develop a media failure sometime ago, but the controller never detected it, because that particular bad area was not read. Your backups worked because they never touched this portion of the disk (ex. empty space, meta data, etc). And then another drive developed a electronics failure, which is instantly detected, putting the array into a degraded mode. When you did a rebuild onto a replace drive, the controller discovered that there was a second failed disk, and this is unrecoverable. 3ware controllers can recover from this situation, all you need to do is tell the controller not to verify the source data. This is a litle dangerous but it has saved me in the past where 1 drive died in a raid 10 array and 2 of the 3 remaining drives had surface defects. The trick was to replace each drive 1 at a time and rebuild without data verification. After 10 painful hours the array was rebuild with out any noticeable data corruption. RAID, of any level, isn't magic. It is important to understand how it works, an realize that drives can passive fail. BTW, if you were using RAID1 or RAID10, you would likely have had the same problem (well, RAID10 can survive _some_ double-disk failures). RAID6 is the only RAID level that can survive failure of any two disks. This is not all true RAID 1 can survive multiple disk failures as it has the storage capacity of 1 spindle and can tolerate the failure of N-1 spindles where N is the number of spindles in the mirror set. This also is kind of true in RAID 10, the more spindles in your mirror sets the more chance you have of being able to survive multiple failures in the array (Say use 6 disks in 2 3 disk mirror sets striped together). The real solution is RAID scrubbing: a low level background process that reads every sector of every disk. All of the real RAID systems do this (usually scheduled weekly, or every other week). Most 3ware RAID card don't have this feature. So rather than not using RAID5 or RAID6 again, you should just not use 3ware anymore. If you use the 3dm2 management interface you can schedule verify and rebuild tasks to run on a regular basis. I think that 7500 series controllers can do this, 9500 and 9550's definitely can. We have 50+ systems that are using 3ware cards (7500-9550 4 and 8 channel models) with 200+ spindles in use (no host spares unfortunately) and drives in that pool failing on average around once a month. We have only ever had trouble recovering from failed drives on 7500 series controllers that have been in production for a reasonably long time. I don't think that you are justified in your slagging off of 3ware controllers. Tom ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Friday 24 August 2007 23:04:37 kirjutas Matthew Dillon: A friend of mine once told me that the only worthwhile RAID systems are the ones that email you a detailed message when something goes south. -Matt ___ AFAIK all good raid hardware producers (3ware, LSI, Areca) got this functionality. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On 8/25/07, Tom Samplonius [EMAIL PROTECTED] wrote: - Artem Kuchin [EMAIL PROTECTED] wrote: ... But i don't understand how and why it happened. ONly 6 hours ago (a night before) all those files were backed up fine w/o any read error. And now, right after replacing the driver and starting rebuild it said that there are bad sectors all over those file. How come? What happened to you was an extremely common occurrence. You had a disk develop a media failure sometime ago, but the controller never detected it, because that particular bad area was not read. Your backups worked because they never touched this portion of the disk (ex. empty space, meta data, etc). And then another drive developed a electronics failure, which is instantly detected, putting the array into a degraded mode. When you did a rebuild onto a replace drive, the controller discovered that there was a second failed disk, and this is unrecoverable. RAID, of any level, isn't magic. It is important to understand how it works, an realize that drives can passive fail. BTW, if you were using RAID1 or RAID10, you would likely have had the same problem (well, RAID10 can survive _some_ double-disk failures). RAID6 is the only RAID level that can survive failure of any two disks. The real solution is RAID scrubbing: a low level background process that reads every sector of every disk. All of the real RAID systems do this (usually scheduled weekly, or every other week). Most 3ware RAID card don't have this feature. So rather than not using RAID5 or RAID6 again, you should just not use 3ware anymore. Tom It's a common mistake to assume that regular maintenance are not required which is implied through It was working for years. Your just waiting for some event to happen which could result in catastrophic failure like this. It's always good to act, rather than react especially dealing with critical data. -Manjunath ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
- David Schwartz [EMAIL PROTECTED] wrote: It is supposed to be for detecting data corruption, so if the card isn't using the checksum, its kinda of useless. You are confused. Checking for data corruption is done, by checking if the *DATA* is corrupt. This does not require looking at the RAID5 checksum since the data has its own data checksum. No, not really. You are just referring to parity as checksums. They are different. Many RAID systems have checksums in addition to parity. For example, Netapp ZCS disks. ... However, in this particular case, validating checksums would have been unhelpful, since the disk was unreadable. diskcheckd would have detected this issue. It would probably have prevented the problem, if it had been running previously. No, it would have saved him. The problem was he lost a drive, and checksums *ON* *OTHER* *DRIVES* were unreadable. Quite possibly they had been unreabable for months, but were never checked, since they are only *needed* to reconstruct the data. Which is what I said? The data on the other disks is unreadable. It doesn't really matter what parity or data was on those sectors. Yes, diskcheckd would only read data sectors. ZFS is also a good option. It has file level checksumming. ZFS never trusts the disks, and is super paranoid. And ZFS can do background scrubbing too. I can't wait for ZFS in FreeBSD 7, because ZFS in software is going to 10 x better than anything 3ware has. That wouuld not have helped him one bit. When the drive failed, the RAID 5 checksums on the other drives still would not have been scrubbed. The RAID 5 checksum (technically an XOR) is only needed to recover the RAID 5 array if a drive (or sector) fails. Ok, you should probably not refer to RAID5 parity as checksums. They are different. Some RAID systems have both. And some do not. ZFS checksums the file level data, which is independent from any RAID5 parity. And yes a media scan would have helped. Tom ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. -- Feargal Reilly, Chief Techie, FBI. PGP Key: 0xBD252C01 (expires: 2006-11-30) Web: http://www.fbi.ie/ | Tel: +353.14988588 | Fax: +353.14988489 Communications House, 11 Sallymount Avenue, Ranelagh, Dublin 6. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. Useful like on the front of the drive bays ;-) The Areca cards have a nice function called drive identify that lights up the selected drive's LED. I think the 3ware cards have it too. -Clay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. Useful like on the front of the drive bays ;-) The Areca cards have a nice function called drive identify that lights up the selected drive's LED. I think the 3ware cards have it too. -Clay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On August 24, 2007 02:31 am Clayton Milos wrote: On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. Useful like on the front of the drive bays ;-) The Areca cards have a nice function called drive identify that lights up the selected drive's LED. I think the 3ware cards have it too. That's only useful if the drive LEDs are configured to correctly match the port. :) We've had custom built servers with 4 drive bays arrive with the LEDs either not connected, or connected to different ports on the RAID controller than the drives were on. Very confusing until we started double-checking the connections when the servers arrived. -- Freddie Cash, LPIC-2 CCNT CCLP Network Support Technician School District 73 (250) 377-HELP [377-4357] [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Feargal Reilly wrote: On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. Most controllers have a function to blink LED so you can discover the mapping. Just about every management app that I know of does (yes, they do exist for FreeBSD from many vendors!), and many also have the feature in their BIOS. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Är precis nyinflyttad i Göteborg, så jag vet inte riktigt var Linnéstaden ligger, själv bor jag vid guldheden ett stenkast från wavrinsky platsen. Förmiddagen passar mig bäst också då jag är upptagen på em :) On 8/24/07, Scott Long [EMAIL PROTECTED] wrote: Feargal Reilly wrote: On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. Most controllers have a function to blink LED so you can discover the mapping. Just about every management app that I know of does (yes, they do exist for FreeBSD from many vendors!), and many also have the feature in their BIOS. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On August 24, 2007 02:31 am Clayton Milos wrote: On Tue, 21 Aug 2007 08:57:22 +0400 Artem Kuchin [EMAIL PROTECTED] wrote: Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. I'm just going to highlight the importance of knowing which physical disk is which on your system. About a year ago I had to replace a hot-swappable disk from an array, but then realised I had no idea which physical disk it was as the map of the disks was rather helpfully *inside* the case. Due to the physical setup, I had no way or removing the cover without first powering down the server - which defeated the whole point of paying extra for hot-swap disks. So yeah, be sure to label your disk bays, but be sure to put those labels somewhere *useful*. -fr. Useful like on the front of the drive bays ;-) The Areca cards have a nice function called drive identify that lights up the selected drive's LED. I think the 3ware cards have it too. That's only useful if the drive LEDs are configured to correctly match the port. :) We've had custom built servers with 4 drive bays arrive with the LEDs either not connected, or connected to different ports on the RAID controller than the drives were on. Very confusing until we started double-checking the connections when the servers arrived. People that do that should be lynched! It's that kind of poor workmanship that would make me never touch their gear again. Then again if I had to discover this when I replaced the wrong drive in a failed array I'd be doing the lynching myself! -Clay ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
A friend of mine once told me that the only worthwhile RAID systems are the ones that email you a detailed message when something goes south. -Matt ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Martin Nilsson wrote: Artem Kuchin wrote: But i don't understand how and why it happened. ONly 6 hours ago (a night before) all those files were backed up fine w/o any read error. And now, right after replacing the driver and starting rebuild it said that there are bad sectors all over those file. How come? That is what patrol read is intended to detect before it is a problem. In a RAID5 array the checksums are only used when reconstructing data, if you have a bad block in a checksum sector it will not be detected until a drive have failed and you try to rebuild the array, unfortunately at that time it is too late... Beware that OS software solutions like diskcheckd will not find this as it only reads the data, not the checksums, it must be done on the controller. That exactly was i was talking about. I don't acess to individual disks behind raid unit, so, i cannot doit. I don't know it controller VERIFY command does it right. If it doesm then i shoudl put it into a cron job and do it on weekly basis. Also, it would halpfull it i could get access to number of left reserved sector for remapping. Any idea about these two for 3ware controllers? Also, someone should mention, that while using raid MUST do verifies often. -- Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On Tue, 21 Aug 2007, Artem Kuchin wrote: could get access to number of left reserved sector for remapping. Any idea about these two for 3ware controllers? Also, someone should mention, that while using raid MUST do verifies often. You can run smartmontools on disks behind 3ware controllers, eg /dev/twe0 -d 3ware,0 -a -o on -S on -m [EMAIL PROTECTED] /dev/twe0 -d 3ware,1 -a -o on -S on -m [EMAIL PROTECTED] -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C signature.asc Description: This is a digitally signed message part.
Re: A little story of failed raid5 (3ware 8000 series)
Artem Kuchin wrote: That exactly was i was talking about. I don't acess to individual disks behind raid unit, so, i cannot doit. I don't know it controller VERIFY command does it right. If it doesm then i shoudl put it into a cron job and do it on weekly basis. Also, it would halpfull it i could get access to number of left reserved sector for remapping. Any idea about these two for 3ware controllers? Also, someone should mention, that while using raid MUST do verifies often. The 3dm2 software (sysutils/3dm port) can dump the SMART data from individual disks. It can also schedule verify and self-test tasks and identify individual drives by blinking the activity light for the drive. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Artem Kuchin unleashed the infinite monkeys on 20/08/2007 23:38 producing: ---SNIP--- But i don't understand how and why it happened. ONly 6 hours ago (a night before) all those files were backed up fine w/o any read error. And now, right after replacing the driver and starting rebuild it said that there are bad sectors all over those file. I've had that happen on a RAID1 setup before. Because all the bad sectors were in different positions the array worked fine until one of the disks failed. As I'm fond of saying at work, RAID is no substitute for regular, tested, backups. Regarding your problem - you can download the (very detailed) manuals for the 3Ware cards and their software from the 3Ware website. A quick scan suggests you need the mediascan command (P48). Newer controllers (9000 series) support scheduling of their equivalent. -- Rob | Oh my God! They killed init! You bastards! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: A little story of failed raid5 (3ware 8000 series)
While we are on the subject: What is the practical difference between VERIFY and REBUILD with regards to a RAID-5 array? My Highpoint RocketRAID 2320 and 2340 cards can be scheduled to perform either verify or rebuild. I currently have them set to verify the arrays weekly. Is that reasonably often? Do I want to rebuild regularly also (or instead of verify)? /Daniel Eriksson ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Darren Pilgrim wrote: Artem Kuchin wrote: That exactly was i was talking about. I don't acess to individual disks behind raid unit, so, i cannot doit. I don't know it controller VERIFY command does it right. If it doesm then i shoudl put it into a cron job and do it on weekly basis. Also, it would halpfull it i could get access to number of left reserved sector for remapping. Any idea about these two for 3ware controllers? Also, someone should mention, that while using raid MUST do verifies often. The 3dm2 software (sysutils/3dm port) can dump the SMART data from individual disks. It can also schedule verify and self-test tasks and identify individual drives by blinking the activity light for the drive. ___ Problems are: 1) how to parse this dump of data? 2) No scheduling is available on 7000,8000 on 4 port models -- Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
You can run smartmontools on disks behind 3ware controllers, eg /dev/twe0 -d 3ware,0 -a -o on -S on -m [EMAIL PROTECTED] /dev/twe0 -d 3ware,1 -a -o on -S on -m [EMAIL PROTECTED] did this: smartctl /dev/twe0 -d 3ware,1 -a for each driver on another server. Two driver are pretty old, the driver on port 2 is less than a month old. However, ALL of the drives have the same values for this 5 Reallocated_Sector_Ct 0x0033 100 100 036Pre-fail Always - 0 How come the number are the same? Even more, what does this 100 mean? 100% of backup sector space is free or just 100 sectors are available? How many total of them in there. Why does it say Pre-fail if it is WAY above the threshold? This data seems to be useless. Now, i did the same for the raid which failed and got me into so many trobles and has bad sectors now (some files are unredable): smartctl /dev/twe0 -d 3ware,0 -A 5 Reallocated_Sector_Ct 0x0033 100 100 036Pre-fail Always - 0 smartctl /dev/twe0 -d 3ware,1 -A 5 Reallocated_Sector_Ct 0x0033 100 100 036Pre-fail Always - 39 smartctl /dev/twe0 -d 3ware,2 -A 5 Reallocated_Sector_Ct 0x0033 100 100 036Pre-fail Always - 9 Now this is BS!!! Agaim accroding to SMART i shoud lookup at VALUE (100) and see if it is below THRES (36). If it is then i am in trouble. No, it does no work this way. Now, if we look at raw number we see 39 for disk1 and 9 for disk 2 For 39 disk1 also 198 Offline_Uncorrectable 0x0010 100 100 000Old_age Offline - 22 1 Raw_Read_Error_Rate 0x000f 058 055 006Pre-fail Always - 170185544 195 Hardware_ECC_Recovered 0x001a 058 055 000Old_age Always - 170185544 7 Seek_Error_Rate 0x000f 087 060 030Pre-fail Always - 524461066 Even for the newly inserted ( 24 hours ago, absulutelly new) driver: 7 Seek_Error_Rate 0x000f 069 060 030Pre-fail Always - 8525167 195 Hardware_ECC_Recovered 0x001a 069 066 000Old_age Always - 8433725 Now, as i undertand the main indication is Offline_Uncorrectable is raw value of it any more than 0 - REPLACE DRIVER ASAP (or maybe it is too late and it is replace driver asap as soon as Reallocated_Sector_Ct 0 ?) Now, what i don't understand is why Hardware_ECC_Recovered and Seek_Error_Rate are so hight. The first one is maybe relate to cabling problem. The driver are all in hot swap baskets of supermicro 2u case. Maybe backpanel is no so good? Seek_Error_Rate is a mistety for me. Any idea? -- Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: A little story of failed raid5 (3ware 8000 series)
While we are on the subject: What is the practical difference between VERIFY and REBUILD with regards to a RAID-5 array? Verify should at a minimum read all the data. Ideally, it would read the checksum blocks too to make sure they are still valid, but it might not. Rebuild should read all the data and write out new checksum blocks. It might also validate that it can read back the checksum. My Highpoint RocketRAID 2320 and 2340 cards can be scheduled to perform either verify or rebuild. I currently have them set to verify the arrays weekly. Is that reasonably often? Do I want to rebuild regularly also (or instead of verify)? It really depends what these functions actually do. Ideally, verify would read the data and the checksum and so would ensure that all data can be read, even if one drive fails. If so, that would really be all you need to do periodically. DS ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
On Tue, 21 Aug 2007, Artem Kuchin wrote: Now, what i don't understand is why Hardware_ECC_Recovered and Seek_Error_Rate are so hight. The first one is maybe relate to cabling problem. The driver are all in hot swap baskets of supermicro 2u case. Maybe backpanel is no so good? Seek_Error_Rate is a mistety for me. Any idea? I don't know what the problem is, I would have expected the drive to report errors in it's log if it is genuinely failing (I've seen this on my laptop) Have you tried running SMART tests on the disk? I'm not saying SMART is the be all and end all of failure monitoring but it has indicated problems to me in the past :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C signature.asc Description: This is a digitally signed message part.
Re: A little story of failed raid5 (3ware 8000 series)
Artem Kuchin wrote: Darren Pilgrim wrote: Artem Kuchin wrote: That exactly was i was talking about. I don't acess to individual disks behind raid unit, so, i cannot doit. I don't know it controller VERIFY command does it right. If it doesm then i shoudl put it into a cron job and do it on weekly basis. Also, it would halpfull it i could get access to number of left reserved sector for remapping. Any idea about these two for 3ware controllers? Also, someone should mention, that while using raid MUST do verifies often. The 3dm2 software (sysutils/3dm port) can dump the SMART data from individual disks. It can also schedule verify and self-test tasks and identify individual drives by blinking the activity light for the drive. ___ Problems are: 1) how to parse this dump of data? Actually smartctl (part of smartmontools) can read the SMART data through a 3ware controller. 2) No scheduling is available on 7000,8000 on 4 port models Use tw_sched then. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Artem Kuchin wrote: So, no raid5 or even raid 6 for me any more. Never! A better policy is to invest in a higher quality RAID controller. Also, always use a battery backup on the controller, and always have an extra disk configured as a hot spare. Data integrity is expensive, unfortunately. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
Artem Kuchin wrote: A day ago at 11 am i have turn off the server, pull out the old driver, installed a new one, turned of the server and started rebuild in an hour from remote location via web interface. After about 5 minuted the machine became unresponsive. Tried rebooting - nothing. I went to the machine and fingure out, that rebuild failed (0%) and some data cannot be read because of bad sectors. We had a similar failure a few years ago, in fact every drive in the array had bad sectors, but one failed completely. Rebuild would not work no matter what we did. Yet another reason to be sure your RAID disks come from different manufacturing batches! - i cannot make it ignore read errors - i cannot figure out which driver has bad sectors (maybe someone know it?) We recovered from that situation like so: 1. Power off the server 2. Pull each drive, attach to another box w/o RAID adapter and use diagnostic tools to run a surface scan/remap on each disk. (These were SCSI disks, not sure if the same applies to PATA/SATA) 3. Put all drives back, including a replacement for the truly failed drive 4. Let the array rebuild Many crossed fingers/toes/eyes later, it came back to life. We replaced the whole box shortly thereafter. The downside was the entire server was offline for the duration of the process, instead of being online during a normal rebuild. So, no raid5 or even raid 6 for me any more. Never! If it's done properly, with hot spares and other failsafe measures, it isn't too bad. Sometimes it's the best available option due to budget/hardware/etc constraints, especially on older systems. RAID can be a tough beast, though. We had one server that ran fine for nearly 5 years on a single PATA disk. Two months after I rebuild it with a proper SCSI RAID setup, it has a multi-drive failure and bombs. Sometimes all the safety measures in the world can't make up for what passes for hardware quality these days... Jim ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: A little story of failed raid5 (3ware 8000 series)
A day ago at 11 am i have turn off the server, pull out the old driver, installed a new one, turned of the server and started rebuild in an hour from remote location via web interface. After about 5 minuted the machine became unresponsive. Tried rebooting - nothing. I went to the machine and fingure out, that rebuild failed (0%) and some data cannot be read because of bad sectors. Why would you power cycle a RAID 5 array with a failed drive? That's like the biggest no-no that there is. When you lose a drive on a RAID 5 array, you are vulnerable until a replacement drive is configured and the array is rebuilt. Any high risk operations during that time would be foolhardy. So, no raid5 or even raid 6 for me any more. Never! Since RAID6 would have saved you from what presumably was a drive failure before a rebuild could be done, it's hard to understand why you would say this is a reason to avoid RAID 6. Perhaps you would do better to understand your failure and avoid the causes of the failure rather than avoiding the things you happened to be using at the time of the failure. If you get food poisoning while wearing a blue shirt, the solution is not to avoid blue shirts in the future. DS ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: A little story of failed raid5 (3ware 8000 series)
David Schwartz wrote: A day ago at 11 am i have turn off the server, pull out the old driver, installed a new one, turned of the server and started rebuild in an hour from remote location via web interface. After about 5 minuted the machine became unresponsive. Tried rebooting - nothing. I went to the machine and fingure out, that rebuild failed (0%) and some data cannot be read because of bad sectors. Why would you power cycle a RAID 5 array with a failed drive? That's like the biggest no-no that there is. When you lose a drive on a RAID 5 array, you are vulnerable until a replacement drive is configured and the array is rebuilt. Any high risk operations during that time would be foolhardy. Um.. it is because i did not have a map of hot swap baskets to conroller ports and i needed to check every driver basket to understand which port it sits on. I have no choise, i think. So, no raid5 or even raid 6 for me any more. Never! Since RAID6 would have saved you from what presumably was a drive failure before a rebuild could be done, it's hard to understand why you would say this is a reason to avoid RAID 6. Perhaps you would do better to understand your failure and avoid the causes of the failure rather than avoiding the things you happened to be using at the time of the failure. If you get food poisoning while wearing a blue shirt, the solution is not to avoid blue shirts in the future. Read the post before. I still don't known which driver has the bad sectors. And it is very posible that EACH driver has them. RAID6 would not help then. -- Artem ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
