On Sunday June 24, [EMAIL PROTECTED] wrote:
> Hi!
>
> Neil Brown wrote:
> > On Thursday June 14, [EMAIL PROTECTED] wrote:
> > >
> > > Dear All
> > >
> > > I've just had a disk (sdc) fail in my raid5 array (sdb sdc sdd),
> >
> > Great! A real live hardware failure1 It is always more satisfying to
> > watch one of those than to have to simulate them all the time!!
> > Unless of course they are fatal... not the case here it seems.
> >
> Well, here comes a _real_ fatal one...
And a very detailed report it was, thanks.
I'm not sure that you want to know this, but it looks like you might
have been able to recover your data.... though it is only a "might".
> Jun 19 09:18:23 wien kernel: (read) scsi/host0/bus0/target0/lun0/part4's sb offset:
>16860096 [events: 00000024]
> Jun 19 09:18:23 wien kernel: (read) scsi/host0/bus0/target1/lun0/part4's sb offset:
>16860096 [events: 00000024]
> Jun 19 09:18:23 wien kernel: (read) scsi/host0/bus0/target2/lun0/part4's sb offset:
>16860096 [events: 00000023]
> Jun 19 09:18:23 wien kernel: (read) scsi/host0/bus0/target3/lun0/part4's sb offset:
>16860096 [events: 00000028]
The reason that this array couldn't restart was that the 4th drive had
the highest event count and it was alone in this. It didn't even have
any valid data!!
Had you unplugged this drive and booted, it would have tried to
assemble an array out of the first two (event count 24). This might
have worked (though it might not, read on).
Alternately, you could have created a raidtab which said that the
third drive was failed, and then run "mkraid"...
mdctl, when it is finished, should be able to make this all much
easier.
But what went wrong? I don't know the whole story but:
- On the first error, the drive was disabled and reconstruction was
started.
- On the second error, the reconstruction was inapprporiately
interrupted. This is an error that I will have to fix in 2.4.
However it isn't really fatal error.
- Things were then going fairly OK, though noisy, until:
> Jun 19 09:10:07 wien kernel: attempt to access beyond end of device
> Jun 19 09:10:07 wien kernel: 08:04: rw=0, want=1788379804, limit=16860217
> Jun 19 09:10:07 wien kernel: dev 09:00 blksize=1024 blocknr=447094950
>sector=-718207696 size=4096 count=1
For some reason, it tried to access well beyond the end of one of the
underlying drives. This caused that drive to fail. This relates to
the subsequent message:
> Jun 19 09:10:07 wien kernel: raid5: restarting stripe 3576759600
which strongly suggests that the filesystem actually asked the raid5
array for a block that was well out of range.
In 2.4, this will be caught before the request gets to raid5. In 2.2
it isn't. The request goes on to raid5, raid5 blindly passes a bad
request down to the disc. The disc reports an error, and raid5
thinks the disc has failed, rather than realise that it never should
have made such a silly request.
But why did the filesystem ask for a block that was out of range?
This is the part that I cannot fathom. It would seem as though the
filesystem got corrupt somehow. Maybe an indirect block got replaced
with garbage, and ext2fs believed the indirect block and went seeking
way off the end of the array. But I don't know how the corruption
happened.
Had you known enough to restart the array from the two apparently
working drives, and then run fsck, it might have fixed things enough
to keep going. Or it might not, depending on how much corruption
there was.
So, Summary of problems:
1/ md responds to a failure on a know-failed drive inappropriately.
This shouldn't be fatal but needs fixing.
2/ md isn't thoughtful enough about updating the event counter on
superblocks and can easily leave an array in an unbuildable
state. This needs to be fix. It's on my list...
3/ raid5 responds to a request for an out-of-bounds device address
by passing on out-of-bounds device addresses the drives, and then
thinking that those drives are failed.
This is fixed in 2.4
4/ Something caused some sort of filesystem corruption. I don't
know what.
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [EMAIL PROTECTED]
