(I've not gotten any replies to my previous two posts - Is this the
right forum for this type of discussion?)
Summary: Commenting out the logic for 'goto retry_write;' seems to
help lessen the impact of losing a disk while reconstruction is going
on.
Again, the trouble I'm noticing is that when a disk fails, the machine
gets hung up spewing out tons of error messages. My reason for the
concern is that the whole reason I'm using raid0 is for high
availability. When the machine loses a disk, it causes the machine
to get stuck in loops like this, and the machine isn't going to be
available until someone goes around to reboot it. I'd much rather
have the system keep running in degraded mode until someone can
intervene.
------------------------------------------------------------------------------
I added more prink's to md.c and raid1.c and invoked a different error
scenario. in md.c for the function cmd_do_sync().
Near the goto reentry point:
retry_write:
printk ("md%d: at retry_write\n", mdidx(mddev));
Near "write error" detection at the bottom of the for loop I just
added on the number of error retries
printk ( KERN_WARNING "md%d: write error while
reconstructing, at block %u(%d) %d errs left.\n",
mdidx(mddev), j, blocksize,
max_write_errors);
This time, I made it hang up during a 'normal' resync instead
of using raidhotadd
$ # the raid set is in degraded mode - only one disk
$ # the disk is powered back on
$ # Reboot the machine, leave the disk power on
$ # after the reboot, start the raid partition with
$ raidstart --configfile /etc/raidtab.md0 /dev/mdo
(the syncronization starts)
(The message 'md0: at retry_write' prints out quite a bit and in my
syslog the last message is being repeated hundreds of times.)
$ # pull the plug on the disk
(unresponsive again)
...
Sep 15 10:26:48 dru1a kernel: SCSI disk error : host 0 channel 0 id 2 lun 0 return
code = 26030000
Sep 15 10:26:48 dru1a kernel: scsidisk I/O error: dev 08:11, sector 772422
Sep 15 10:26:48 dru1a kernel: raid1: request to stop resync
Sep 15 10:26:48 dru1a kernel: interrupting MD-thread pid 6 mdrecoveryd
Sep 15 10:26:48 dru1a kernel: raid1: total working disks is 1. disk[0].operational=1
disk[1].operational=0
Sep 15 10:26:48 dru1a kernel: raid1: only one disk left and IO error.
Sep 15 10:26:48 dru1a kernel: SCSI disk error : host 0 channel 0 id 2 lun 0 return
code = 26030000
Sep 15 10:26:48 dru1a kernel: scsidisk I/O error: dev 08:11, sector 772550
Sep 15 10:26:48 dru1a kernel: raid1: request to stop resync
Sep 15 10:26:48 dru1a kernel: interrupting MD-thread pid 6 mdrecoveryd
Sep 15 10:26:48 dru1a kernel: raid1: total working disks is 1. disk[0].operational=1
disk[1].operational=0 n
Sep 15 10:26:48 dru1a kernel: raid1: only one disk left and IO error.
What's interesting about the error messages is that the 'retry_write'
stops coming out, so my assumption is that the 'recovery' thread isn't
actually adding writes to the queue.
here's the output of ps
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.3 0.3 1120 476 ? S 09:52 0:06 init [3]
root 2 0.0 0.0 0 0 ? SW 09:52 0:00 [kflushd]
root 3 0.0 0.0 0 0 ? SW 09:52 0:00 [kupdate]
root 4 0.0 0.0 0 0 ? SW 09:52 0:00 [kpiod]
root 5 0.0 0.0 0 0 ? SW 09:52 0:00 [kswapd]
root 6 1.0 0.0 0 0 ? DWN 09:52 0:20 [mdrecoveryd]
Notably, state 'D' is uninterruptible sleep, so the recovery thread
is probably blocking on some system call...
After a while , I type '/sbin/reboot'. nothing happens, but I keep a
tail on the messages long and my patience pays off: A different error
comes out!!!
Sep 15 10:29:03 dru1a kernel: BAD WRITE 386176>
Sep 15 10:29:03 dru1a kernel: md0: write error while reconstructing, at block
386176(1024) 4093 errs left.
Sep 15 10:29:03 dru1a kernel: md0: at retry_write
And all of the sudden the shutdown sequence starts to have an effect,
but then it hangs again. Attaching through the serial port to the
console shows that the same old monkeyshine is going on...
# grep "reconstructing" messages
Sep 15 10:18:46 dru1a kernel: md0: write error while reconstructing, at block
386176(1024) 4095 errs left.
Sep 15 10:23:54 dru1a kernel: md0: write error while reconstructing, at block
386176(1024) 4094 errs left.
Sep 15 10:29:03 dru1a kernel: md0: write error while reconstructing, at block
386176(1024) 4093 errs left.
It looks like it takes 5 minutes between each loop. At this rate, it
will take 4093 * 5 minutes before I get my machine back through the logic:
printk( KERN_ALERT "too many write errors, stopping reconstruction.\n");
-------------------------------------------------------------------------------
It appears that the SCSI driver (midlevel? low level) is retrying
these writes a couple of times. So why does the 'md' driver need
to retry them?
in md.c in the recovery thread, I commented out the retry of the write:
#if 0
if (max_write_errors) {
max_write_errors--;
/* MOD BY EZA 09/15/00 */
printk ( KERN_WARNING "md%d: write error while
reconstructing, at block %u(%d) %d errs left.\n",
mdidx(mddev), j, blocksize,
max_write_errors);
goto retry_write;
}
#endif
and tried the test again. After about 7 minutes, the reconstruction
stopped and I got the machine back:
Sep 15 11:52:14 dru1a kernel: md: md0: raid array is not clean -- starting background
reconstruction
Sep 15 11:52:15 dru1a kernel: md: minimum _guaranteed_ reconstruction speed: 100
KB/sec.
Sep 15 11:52:15 dru1a kernel: md: using maximum available idle IO bandwith for
reconstruction.
... lots of messages ...
Sep 15 11:59:17 dru1a kernel: too many write errors, stopping reconstruction.
Sep 15 11:59:18 dru1a kernel: mdrecoveryd: md0: no spare disk to reconstruct array! --
continuing in degraded mode
Sep 15 11:59:18 dru1a kernel: mdrecoveryd: md0: no spare disk to reconstruct array! --
continuing in degraded mode
Hooray! I survived a disk failure! I'm running in degraded mode!
-------------------------------------------------------------------------------
It looks like 64 different writes failed before the recovery thread
got any indication of there being an error. I have 64 instances of:
Sep 15 11:57:32 dru1a kernel: scsidisk I/O error: dev 08:11, sector 390350
Sep 15 11:57:36 dru1a kernel: scsidisk I/O error: dev 08:11, sector 390352
Sep 15 11:57:40 dru1a kernel: scsidisk I/O error: dev 08:11, sector 390354
Sep 15 11:57:45 dru1a kernel: scsidisk I/O error: dev 08:11, sector 390356
...
-------------------------------------------------------------------------------
Also, the SCSI bus is reset 128 times (I guess this is an issue for the
Adaptec driver or scsi driver in general.)
Of course, I wish I could eliminate the 5 minute delay before
detecting the write error, but I can live with the system being out of
action for 5 minutes.
-Eric.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [EMAIL PROTECTED]
