On 08/25/05 23:53, Tejun Heo wrote:
> Hello, fellow SCSI/ATA developers.
>
> This is the first draft of SCSI EH document. This document tries to
> describe how SCSI EH works and what choirs should be done to maintain
> SCSI midlayer integrity. It's intended that this document can be used
> as reference for implementing either fine-grained EH callbacks or
> single eh_strategy_handler() callback.
Very good stuff, Tejun!
I'll have to print it and read it. At first glance, good job!
Thanks,
Luben
> I'm pretty sure that I've screwed up in (hopefully) several places,
> so please correct me. Also, I have several places where I'm not sure
> or have questions, those are marked with *VERIFY* and *QUESTION*
> respectively. If you know the answer, please let me know.
>
> Thanks.
>
>
> SCSI EH
> ======================================
>
> TABLE OF CONTENTS
>
> [1] How SCSI commands travel through the midlayer and to EH
> [1-1] struct scsi_cmnd
> [1-2] How do scmd's get completed?
> [1-2-1] Completing a scmd w/ scsi_done
> [1-2-2] Completing a scmd w/ timeout
> [1-3] How EH takes over
> [2] How SCSI EH works
> [2-1] EH through fine-grained callbacks
> [2-1-1] Overview
> [2-1-2] Flow of scmds through EH
> [2-1-3] Flow of control
> [2-2] EH through hostt->eh_strategy_handler()
> [2-2-1] Pre hostt->eh_strategy_handler() SCSI midlayer conditions
> [2-2-2] Post hostt->eh_strategy_handler() SCSI midlayer conditions
> [2-2-3] Things to consider
>
>
> [1] How SCSI commands travel through the midlayer and to EH
>
> [1-1] struct scsi_cmnd
>
> Each SCSI command is represented with struct scsi_cmnd (== scmd). A
> scmd has two list_head's to link itself into lists. The two are
> scmd->list and scmd->eh_entry. The former is used for free list or
> per-device allocated scmd list and not of much interest to this EH
> discussion. The latter is used for completion and EH lists.
>
>
> [1-2] How do scmd's get completed?
>
> Once LLDD gets hold of a scmd, either the LLDD will complete the
> command by calling scsi_done callback passed from midlayer when
> invoking hostt->queuecommand() or SCSI midlayer will time it out.
>
>
> [1-2-1] Completing a scmd w/ scsi_done
>
> For all non-EH commands, scsi_done() is the completion callback. It
> does the following.
>
> 1. Delete timeout timer. If it fails, it means that timeout timer
> has expired and is going to finish the command. Just return.
>
> 2. Link scmd to per-cpu scsi_done_q using scmd->en_entry
>
> 3. Raise SCSI_SOFTIRQ
>
> SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to
> determine what to do with the command. scsi_decide_disposition()
> looks at the scmd->result value and sense data to determine what to do
> with the command.
>
> - SUCCESS
> scsi_finish_command() is invoked for the command. The
> function does some maintenance choirs and notify completion by
> calling scmd->done() callback, which, for fs requests, would
> be HLD completion callback - sd:sd_rw_intr, sr:rw_intr,
> st:st_intr.
>
> - NEEDS_RETRY
> - ADD_TO_MLQUEUE
> scmd is requeued to blk queue.
>
> - otherwise
> scsi_eh_scmd_add(scmd, 0) is invoked for the command. See
> [1-3] for details of this funciton.
>
>
> [1-2-2] Completing a scmd w/ timeout
>
> The timeout handler is scsi_times_out(). When a timeout occurs, this
> function
>
> 1. invokes optional hostt->eh_timedout() callback. Return value can
> be one of
>
> - EH_HANDLED
> This indicates that eh_timedout() dealt with the timeout. The
> scmd is passed to __scsi_done() and thus linked into per-cpu
> scsi_done_q. Normal command completion described in [1-2-1]
> follows.
>
> - EH_RESET_TIMER
> This indicates that more time is required to finish the
> command. Timer is restarted. This action is counted as a
> retry and only allowed scmd->allowed + 1(!) times. Once the
> limit is reached, EH_NOT_HANDLED action is taken.
>
> *NOTE* This action is racy as the LLDD could finish the scmd
> after the timeout has expired but before it's added back. In
> such cases, scsi_done() would think that timeout has occurred
> and return without doing anything. We lose completion and the
> command will time out again.
>
> - EH_NOT_HANDLED
> This is the same as when eh_timedout() callback doesn't exist.
> Step #2 is taken.
>
> 2. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the
> command. See [1-3] for more information.
>
>
> [1-3] How EH takes over
>
> scmds enter EH via scsi_eh_scmd_add(), which does the following.
>
> 1. Turns on scmd->eh_eflags as requested. It's 0 for error
> completions and SCSI_EH_CANCEL_CMD for timeouts.
>
> 2. Links scmd->eh_entry to shost->eh_cmd_q
>
> 3. Sets SHOST_RECOVERY bit in shost->shost_state
>
> 4. Increments shost->host_failed
>
> 5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed
>
> As can be seen above, once any scmd is added to shost->eh_cmd_q,
> SHOST_RECOVERY shost_state bit is turned on. This prevents any new
> scmd to be issued from blk queue to the host; eventually, all scmds on
> the host either complete normally, fail and get added to eh_cmd_q, or
> time out and get added to shost->eh_cmd_q.
>
> If all scmds either complete or fail, the number of in-flight scmds
> becomes equal to the number of failed scmds - i.e. shost->host_busy ==
> shost->host_failed. This wakes up SCSI EH thread. So, once woken up,
> SCSI EH thread can expect that all in-flight commands have failed and
> are linked on shost->eh_cmd_q.
>
> Note that this does not mean lower layers are quiescent. If a LLDD
> completes a scmd with error status, the LLDD and lower layers are
> assumed to forget about the scmd at that point. However, if a scmd
> has timed out, unless hostt->eh_timedout() made lower layers forget
> about the scmd, which currently no LLDD does, the command is still
> active as long as lower layers are concerned and completion could
> occur at any time. Of course, all such completions are ignored as the
> timer has already expired.
>
> We'll talk about how SCSI EH takes actions to abort - make LLDD
> forget about - timed out scmds later.
>
>
> [2] How SCSI EH works
>
> LLDD's can implement SCSI EH actions in one of the following two
> ways.
>
> - Fine-grained EH callbacks
> LLDD can implement fine-grained EH callbacks and let SCSI
> midlayer drive error handling and call appropriate callbacks.
> This will be dicussed further in [2-1].
>
> - eh_strategy_handler() callback
> This is one big callback which should do whole error handling.
> As such, it should do all choirs SCSI midlayer performs during
> recovery. This will be discussed in [2-2].
>
> Once recovery is complete, SCSI EH resumes normal operation by
> calling scsi_restart_operations(), which
>
> 1. Checks if door locking is needed and locks door.
>
> 2. Clears SHOST_RECOVERY shost_state bit
>
> 3. Wakes up waiters on shost->host_wait. This occurs if someone
> calls scsi_block_when_processing_errors() on the host.
> (*QUESTION* why is it needed? All operations will be blocked
> anyway after it reaches blk queue.)
>
> 4. Kicks queues in all devices on the host in the asses
>
>
> [2-1] EH through fine-grained callbacks
>
> [2-1-1] Overview
>
> If eh_strategy_handler() is not present, SCSI midlayer takes charge
> of driving error handling. EH's goals are two - make LLDD, host and
> device forget about timed out scmds and make them ready for new
> commands. A scmd is said to be recovered if the scmd is forgotten by
> lower layers and lower layers are ready to process or fail the scmd
> again.
>
> To achieve these goals, EH performs recovery actions with increasing
> severity. Some actions are performed by issueing SCSI commands and
> others are performed by invoking one of the following fine-grained
> hostt EH callbacks. Callbacks may be omitted and omitted ones are
> considered to fail always.
>
> int (* eh_abort_handler)(struct scsi_cmnd *);
> int (* eh_device_reset_handler)(struct scsi_cmnd *);
> int (* eh_bus_reset_handler)(struct scsi_cmnd *);
> int (* eh_host_reset_handler)(struct scsi_cmnd *);
>
> Higher-severity actions are taken only when lower-severity actions
> cannot recover some of failed scmds. Also, note that failure of the
> highest-severity action means EH failure and results in offlining of
> all unrecovered devices.
>
> During recovery, the following rules are followed
>
> - Recovery actions are performed on failed scmds on the to do list,
> eh_work_q. If a recovery action succeeds for a scmd, recovered
> scmds are removed from eh_work_q.
>
> Note that single recovery action on a scmd can recover multiple
> scmds. e.g. resetting a device recovers all failed scmds on the
> device.
>
> - Higher severity actions are taken iff eh_work_q is not empty after
> lower severity actions are complete.
>
> - EH reuses failed scmds to issue commands for recovery. For
> timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd
> before reusing it for EH commands.
>
> When a scmd is recovered, the scmd is moved from eh_work_q to EH
> local eh_done_q using scsi_eh_finish_cmd(). After all scmds are
> recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to
> either retry or error-finish (notify upper layer of failure) recovered
> scmds.
>
> scmds are retried iff its sdev is still online (not offlined during
> EH), REQ_FAILFAST is not set and ++scmd->retries is less than
> scmd->allowed.
>
>
> [2-1-2] Flow of scmds through EH
>
> 1. Error completion / time out
> ACTION: scsi_eh_scmd_add() is invoked for scmd
> - set scmd->eh_eflags
> - add scmd to shost->eh_cmd_q
> - set SHOST_RECOVERY
> - shost->host_failed++
> LOCKING: shost->host_lock
>
> 2. EH starts
> ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q
> is cleared.
> LOCKING: shost->host_lock (*VERIFY* not necessary, just for
> consistency)
>
> 3. scmd recovered
> ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd
> - shost->host_failed--
> - clear scmd->eh_eflags
> - scsi_setup_cmd_retry()
> - move from local eh_work_q to local eh_done_q
> LOCKING: none
>
> 4. EH completes
> ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper
> layer of failure.
> - scmd is removed from eh_done_q and scmd->eh_entry is cleared
> - if retry is necessary, scmd is requeued using
> scsi_queue_insert()
> - otherwise, scsi_finish_command() is invoked for scmd
> LOCKING: queue or finish function performs appropriate locking
>
>
> [2-1-3] Flow of control
>
> EH through fine-grained callbacks start from scsi_unjam_host().
>
> <<scsi_unjam_host>>
>
> 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local
> eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is
> cleared by this action.
>
> 2. Invoke scsi_eh_get_sense.
>
> <<scsi_eh_get_sense>>
>
> This action is taken for each error-completed
> (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most
> SCSI transports/LLDDs automatically acquire sense data on
> command failures (autosense). Autosense is recommended as
> sense information could get out of sync inbetween occurrence
> of CHECK CONDITION and this action.
>
> Note that if autosense is not supported, scmd->sense_buffer
> contains invalid sense data when error-completing the scmd
> with scsi_done(). scsi_decide_disposition() always returns
> FAILED in such cases thus invoking SCSI EH. When the scmd
> reaches here, sense data is acquired and
> scsi_decide_disposition() is called again.
>
> 1. Invoke scsi_request_sense() which issues REQUEST_SENSE
> command. If fails, no action. Note that taking no action
> causes higher-severity recovery to be taken for the scmd.
>
> 2. Invoke scsi_decide_disposition() on the scmd
>
> - SUCCESS
> scmd->retries is set to scmd->allowed preventing
> scsi_eh_flush_done_q() from retrying the scmd and
> scsi_eh_finish_cmd() is invoked.
>
> - NEEDS_RETRY
> scsi_eh_finish_cmd() invoked
>
> - otherwise
> No action.
>
> 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
>
> <<scsi_eh_abort_cmds>>
>
> This action is taken for each timed out commands.
> hostt->eh_abort_handler() is invoked for each scmd. The
> handler returns SUCCESS if it has succeeded to make LLDD and
> all related hardware forget about the scmd.
>
> If a timedout scmd is successfully aborted and the sdev is
> either offline or ready, scsi_eh_finish_cmd() is invoked for
> the scmd. Otherwise, the scmd is left in eh_work_q for
> higher-severity actions.
>
> Note that both offline and ready status mean that the sdev is
> ready to process new scmds, where processing implies immediate
> failing; thus, if a sdev is in one of the two states, no
> further recovery action is needed.
>
> Device readiness is tested with scsi_eh_tur() which issues
> TEST_UNIT_READY command. Note that the scmd must be aborted
> successfully before reusing it to issue TEST_UNIT_READY.
>
> 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
>
> <<scsi_eh_ready_devs>>
>
> This function takes four increasingly more severe measures to
> make sdevs with failed scmds ready for new commands.
>
> 1. Invoke scsi_eh_stu()
>
> <<scsi_eh_stu>>
>
> For each sdev which has failed scmds with valid sense data
> of which scsi_check_sense()'s verdict is FAILED,
> START_STOP_UNIT command is issued w/ start=1. Note that
> as we explicitly choose error-completed scmds, it is known
> that lower layers have forgotten about the scmd and we can
> resuse it for STU.
>
> If STU succeeds and the sdev is either offline or ready,
> all failed scmds on the sdev are EH-finished with
> scsi_eh_finish_cmd().
>
> *QUESTION* If hostt->eh_abort_handler() isn't implemented
> or failed, we may still have timed out scmds at this point
> and STU doesn't make lower layers forget about those
> scmds. Yet, we EH-finish all scmds on the sdev if STU
> succeeds. IMHO, we should EH-finish only error-completed
> scmds on the sdev.
>
> 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset().
>
> <<scsi_eh_bus_device_reset>>
>
> This action is very similar to scsi_eh_stu() except that,
> instead of issuing STU, hostt->eh_device_reset_handler()
> is used. Also, as we're not issuing SCSI commands and
> resetting clears all scmds on the sdev, there is no need
> to choose error-completed scmds.
>
> 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset()
>
> <<scsi_eh_bus_reset>>
>
> hostt->eh_bus_reset_handler() is invoked for each channel
> with failed scmds. If bus reset succeeds, all failed
> scmds on all ready or offline sdevs on the channel are
> EH-finished.
>
> 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset()
>
> <<scsi_eh_host_reset>>
>
> This is the last resort. hostt->eh_host_reset_handler()
> is invoked. If host reset succeeds, all failed scmds on
> all ready or offline sdevs are EH-finished.
>
> 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs()
>
> <<scsi_eh_offline_sdevs>>
>
> Take all sdevs which still have unrecovered scmds offline
> and EH-finish the scmds.
>
> 5. Invoke scsi_eh_flush_done_q().
>
> <<scsi_eh_flush_done_q>>
>
> At this point all scmds are recovered (or given up) and
> put on eh_done_q by scsi_eh_finish_cmd(). This function
> flushes eh_done_q by either retrying or notifying upper
> layer of failure of the scmds.
>
>
> [2-2] EH through hostt->eh_strategy_handler()
>
> hostt->eh_strategy_handler() is invoked in the place of
> scsi_unjam_host(). As such, it is responsible for whole recovery
> process. On completion, the handler should have made lower layers
> forget about all failed scmds and ready for new commands or offline.
> Also, it should perform SCSI EH maintenance choirs to maintain
> integrity of SCSI midlayer. IOW, of the steps described in [2-1-2],
> all steps except for #1 must be implemented by eh_strategy_handler().
>
>
> [2-2-1] Pre hostt->eh_strategy_handler() SCSI midlayer conditions
>
> The following conditions are true on entry to the handler.
>
> - Each failed scmd's eh_flags field is set appropriately.
>
> - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry.
>
> - SHOST_RECOVERY is set
>
> - shost->host_failed == shost->host_busy
>
>
> [2-2-2] Post hostt->eh_strategy_handler() SCSI midlayer conditions
>
> The following conditions must be true on exit from the handler.
>
> - shost->host_failed is zero.
>
> - Each scmd's eh_eflags field is cleared.
>
> - Each scmd is in such a state that scsi_setup_cmd_retry() on the
> scmd doesn't make any difference.
>
> - shost->eh_cmd_q is cleared.
>
> - Each scmd->eh_entry is cleared. (*VERIFY* This is currently not
> necessary for correct operation, but keep them cleared anyway for
> consistency.)
>
> - Either scsi_queue_insert() or scsi_finish_command() is called on
> each scmd. Note that scmd->retries and ->allowed can be used to
> limit the number of retries.
>
>
> [2-2-3] Things to consider
>
> - Know that timed out scmds are still active on lower layers. Make
> lower layers forget about them before doing anything else with
> those scmds.
>
> - For consistency, when accessing/modifying shost data structure,
> grab shost->host_lock.
>
> - On completion, each failed sdev must have forgotten about all
> active scmds.
>
> - On completion, each failed sdev must be ready for new commands or
> offline.
>
>
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
