On Monday, I lost power to the machine. (Power cord yanked out, not power failure.) I powered the machine back on, and it started up without issues. About an hour later, I got emails from mdadm saying that 2 of my 4 RAID devices had failed.
I assumed disk had not actually physically failed, since 2 of my 4 devices were still online and (apparently) fine. I removed the failed partitions from their arrays, and they re-added them. mdadm happily re-synced them, and everything seemed fine. Now, a few days later, the same 2 partitions (md0, md1) are again saying they have failed, and md3 has also failed. The failures didn't happen at the same time.
I would appreciate some help troubleshooting here. What else should I do to investigate the problem partitions, or the disk itself? Prior to this little poweroff incident, this machine had been running without any issues for several months. I'm using an old workstation with a cheapo IDE SATA controller. (Maybe that's the problem?)
*** Timeline *** 7/13 18:00 (roughly) : power yanked 7/13 22:00 (roughly) : power restored 7/14 9:24 : email from mdadm. sda5 (md1) failed. 7/14 9:29 : email from mdadm. sda1 (md0) failed.7/15 morning (can't find the log entry to get exact time) : remove/add sda5 and sda1. resynced w/o issues.
7/15 20:20 : email from mdadm. sda1 (md0) failed. 7/16 0:07 : email from mdadm. sda7 (md3) failed. 7/16 9:38 : email from mdadm. sda5 (md1) failed. *** Syslog Errors ***I notice errors like this in /var/log/syslog in a few places. I can try to track them all down if that's helpful.
Jul 14 09:24:29 gringotts kernel: [50073.107428] ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen Jul 14 09:24:29 gringotts kernel: [50073.107509] ata3: SError: { PHYRdyChg } Jul 14 09:24:29 gringotts kernel: [50073.107550] ata3.00: cmd ea/ 00:00:00:00:00/00:00:00:00:00/a0 tag 0 Jul 14 09:24:29 gringotts kernel: [50073.107553] res d0/ d0:d0:d0:d0:d0/00:00:00:00:00/d0 Emask 0x12 (ATA bus error) Jul 14 09:24:29 gringotts kernel: [50073.107639] ata3.00: status: { Busy } Jul 14 09:24:29 gringotts kernel: [50073.107670] ata3.00: error: { ICRC UNC IDNF } Jul 14 09:24:29 gringotts kernel: [50073.107714] ata3: hard resetting link Jul 14 09:24:35 gringotts kernel: [50078.590072] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Jul 14 09:24:35 gringotts kernel: [50078.651141] ata3.00: configured for UDMA/100 Jul 14 09:24:35 gringotts kernel: [50078.651190] end_request: I/O error, dev sda, sector 58604927 Jul 14 09:24:35 gringotts kernel: [50078.651244] md: super_written gets error=-5, uptodate=0 Jul 14 09:24:35 gringotts kernel: [50078.651257] raid1: Disk failure on sda5, disabling device. Jul 14 09:24:35 gringotts kernel: [50078.651260] raid1: Operation continuing on 1 devices.
Jul 14 09:24:35 gringotts kernel: [50078.651353] ata3: EH completeJul 14 09:24:35 gringotts kernel: [50078.653030] sd 2:0:0:0: [sda] 1250263728 512-byte hardware sectors (640135 MB) Jul 14 09:24:35 gringotts kernel: [50078.664063] sd 2:0:0:0: [sda] Write Protect is off Jul 14 09:24:35 gringotts kernel: [50078.664080] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 Jul 14 09:24:35 gringotts kernel: [50078.664184] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 14 09:24:35 gringotts kernel: [50078.677073] RAID1 conf printout: Jul 14 09:24:35 gringotts kernel: [50078.677094] --- wd:1 rd:2Jul 14 09:24:35 gringotts kernel: [50078.677103] disk 0, wo:1, o:0, dev:sda5 Jul 14 09:24:35 gringotts kernel: [50078.677109] disk 1, wo:0, o:1, dev:sdb5
Jul 14 09:24:35 gringotts kernel: [50078.679371] RAID1 conf printout: Jul 14 09:24:35 gringotts kernel: [50078.679389] --- wd:1 rd:2Jul 14 09:24:35 gringotts kernel: [50078.679399] disk 1, wo:0, o:1, dev:sdb5 Jul 14 09:24:35 gringotts mdadm[3495]: Fail event detected on md device /dev/md1, component device /dev/sda5
Jul 14 19:02:05 gringotts kernel: [84728.741110] ata3: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen Jul 14 19:02:05 gringotts kernel: [84728.741190] ata3: SError: { PHYRdyChg } Jul 14 19:02:05 gringotts kernel: [84728.741234] ata3: hard resetting link Jul 14 19:02:11 gringotts kernel: [84734.530052] ata3: link is slow to respond, please be patient (ready=-19) Jul 14 19:02:15 gringotts kernel: [84738.790046] ata3: COMRESET failed (errno=-16) Jul 14 19:02:15 gringotts kernel: [84738.790108] ata3: hard resetting link Jul 14 19:02:16 gringotts kernel: [84739.600066] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Jul 14 19:02:16 gringotts kernel: [84739.660599] ata3.00: configured for UDMA/100
Jul 14 19:02:16 gringotts kernel: [84739.660627] ata3: EH completeJul 14 19:02:16 gringotts kernel: [84739.661665] sd 2:0:0:0: [sda] 1250263728 512-byte hardware sectors (640135 MB) Jul 14 19:02:16 gringotts kernel: [84739.662300] sd 2:0:0:0: [sda] Write Protect is off Jul 14 19:02:16 gringotts kernel: [84739.662315] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 Jul 14 19:02:16 gringotts kernel: [84739.663229] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jul 16 09:38:16 gringotts kernel: [223700.170900] ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen Jul 16 09:38:16 gringotts kernel: [223700.170985] ata3: SError: { PHYRdyChg } Jul 16 09:38:16 gringotts kernel: [223700.171029] ata3.00: cmd ea/ 00:00:00:00:00/00:00:00:00:00/a0 tag 0 Jul 16 09:38:16 gringotts kernel: [223700.171033] res d0/ d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error) Jul 16 09:38:16 gringotts kernel: [223700.171120] ata3.00: status: { Busy } Jul 16 09:38:16 gringotts kernel: [223700.171151] ata3.00: error: { ICRC UNC IDNF } Jul 16 09:38:16 gringotts kernel: [223700.171197] ata3: hard resetting link Jul 16 09:38:22 gringotts kernel: [223705.600073] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Jul 16 09:38:22 gringotts kernel: [223705.660564] ata3.00: configured for UDMA/100 Jul 16 09:38:22 gringotts kernel: [223705.660609] end_request: I/O error, dev sda, sector 58604927 Jul 16 09:38:22 gringotts kernel: [223705.660667] md: super_written gets error=-5, uptodate=0 Jul 16 09:38:22 gringotts kernel: [223705.660678] raid1: Disk failure on sda5, disabling device. Jul 16 09:38:22 gringotts kernel: [223705.660682] raid1: Operation continuing on 1 devices.
Jul 16 09:38:22 gringotts kernel: [223705.660771] ata3: EH completeJul 16 09:38:22 gringotts kernel: [223705.660902] sd 2:0:0:0: [sda] 1250263728 512-byte hardware sectors (640135 MB) Jul 16 09:38:22 gringotts kernel: [223705.660954] sd 2:0:0:0: [sda] Write Protect is off Jul 16 09:38:22 gringotts kernel: [223705.660962] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00 Jul 16 09:38:22 gringotts kernel: [223705.661043] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
*** Background Info *** r...@gringotts:~# uname -aLinux gringotts 2.6.27-11-server #1 SMP Thu Jan 29 20:19:41 UTC 2009 i686 GNU/Linux
r...@gringotts:~# cat /proc/mdstatPersonalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sda7[2](F) sdb7[1]
593874752 blocks [2/1] [_U]
md2 : active raid1 sda6[0] sdb6[1]
1951744 blocks [2/2] [UU]
md1 : active raid1 sda5[2](F) sdb5[1]
19534912 blocks [2/1] [_U]
md0 : active raid1 sda1[2](F) sdb1[1]
9767424 blocks [2/1] [_U]
unused devices: <none>
r...@gringotts:~# mount
/dev/md0 on / type ext3 (rw,relatime,errors=remount-ro)
/dev/md1 on /home type ext3 (rw,relatime,user_xattr)
/dev/md3 on /opt/backup type ext3 (rw,relatime,user_xattr)
NOTE /dev/md2 is swap space.
r...@gringotts:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Sun Feb 22 10:07:22 2009
Raid Level : raid1
Array Size : 9767424 (9.31 GiB 10.00 GB)
Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Jul 16 10:20:08 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
UUID : e002f922:2f6ba792:a34321e9:b82a201b
Events : 0.10924
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 17 1 active sync /dev/sdb1
2 8 1 - faulty spare /dev/sda1
r...@gringotts:~# mdadm --detail /dev/md1
/dev/md1:
Version : 00.90
Creation Time : Sun Feb 22 10:07:32 2009
Raid Level : raid1
Array Size : 19534912 (18.63 GiB 20.00 GB)
Used Dev Size : 19534912 (18.63 GiB 20.00 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Thu Jul 16 10:20:21 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
UUID : 799faf28:c7ea5c83:21a2963b:0dddc562
Events : 0.1602
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 21 1 active sync /dev/sdb5
2 8 5 - faulty spare /dev/sda5
r...@gringotts:/var/log# mdadm --detail /dev/md2
/dev/md2:
Version : 00.90
Creation Time : Sun Feb 22 10:07:40 2009
Raid Level : raid1
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 1951744 (1906.32 MiB 1998.59 MB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Thu Jul 16 07:23:32 2009
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : d78f4817:88606cc2:cd273dbb:85d5ca20
Events : 0.18
Number Major Minor RaidDevice State
0 8 6 0 active sync /dev/sda6
1 8 22 1 active sync /dev/sdb6
r...@gringotts:/var/log# mdadm --detail /dev/md3
/dev/md3:
Version : 00.90
Creation Time : Sun Feb 22 10:07:47 2009
Raid Level : raid1
Array Size : 593874752 (566.36 GiB 608.13 GB)
Used Dev Size : 593874752 (566.36 GiB 608.13 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 3
Persistence : Superblock is persistent
Update Time : Thu Jul 16 01:43:28 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
UUID : 44f192d1:72c1a98b:1027f96a:5aea9347
Events : 0.2524
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 23 1 active sync /dev/sdb7
2 8 7 - faulty spare /dev/sda7
PGP.sig
Description: This is a digitally signed message part
--------------------------------------------------- PLUG-discuss mailing list - [email protected] To subscribe, unsubscribe, or to change your mail settings: http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
