Hello, Jeff, Albert & ATA developers.
This is the final one of recent document series for libata EH - SCSI
EH, ATA exceptions, libata EH and, this one - libata new EH.
This document tries to discuss how to implement new advanced EH. It
also describes some proposed mechanisms in detail. I'm aware that
things are vague without actual code, but I still think this document
alone can at least help discussion if nothing else. As long as some
consensus is reached regarding general desing, I'll follow up with
patches.
Jeff, a lot are from my previous new EH/NCQ patchset but also quite
a bit has changed (for better, I hope).
Thanks.
libata new EH
======================================
As discussed in the previous libata EH doc, the current libata EH
needs some improvements. This document discusses goals of new libata
EH and how to reach them. Please read SCSI EH, ATA exceptions and
libata EH documents first.
TABLE OF CONTENTS
[1] Goals & design choices
[1-1] Use SCSI hostt->eh_strategy_handler()
[1-2] Unified error path in an EH thread
[1-3] Synchronization
[1-4] Clean mechanism to hand off qc's to EH
[1-5] Separate EH qc
[1-6] SCSI/libata separation
[2] Designs
[2-1] Handoff of failed qc's
[2-2] Timed out scmd's and qc's
[2-3] Summary of [2-1] and [2-2]
[2-4] EH processing & completion
[3] Ideas
[3-1] Using EH for non-error exceptions and dynamic reconfiguration
[3-2] Using EH for host_set level exclusion
[4] Implementation plan
[1] Goals & design choices
The final goal is implementing advanced error handling as described
in ATA exceptions document including NCQ EH, dynamic transport
reconfiguration and non-error exception handling for power management
and hot plugging.
The followings are sub goals and design choices to reach the final
goal.
[1-1] Use SCSI hostt->eh_strategy_handler()
We have two other alternatives here - one is using fine-grained
SCSI EH callbacks and the other is implementing separate EH for
libata.
Using fine-grained SCSI EH callbacks is possible, but it has too
much SCSI/SPI assumptions in it - ATA error handling can be quite
different from SCSI error handling. Also, as described in the
SCSI EH doc, it issues several SCSI commands for recovery. They
can be translated but recovery through translation is a bit
creepy, IMHO.
The second option - private EH implementation - is attractive in
that it will be better integrated into libata. However,
implementing a full EH when a generic framwork is already in place
doesn't make a lot of people happy. And, I think integration
problems can be worked around without too much trouble.
The basic semantics of eh_strategy_handler() are
- Full context EH.
- After EH is started, all normal command processing is suspended
until EH is complete.
- Once EH is determined to be necessary, active commands are
drained by suppressing all command issuing and waiting for
in-flight commands. When EH is finally entered, all active
commands are failed commands.
IMO, above semantics are fairly fundamental to block device error
handling and, in the future, to whatever framework libata
migrates, assuming above semantics shouldn't hurt too much.
[1-2] Unified error path in an EH thread
Currently EH is scattered around several places including the
interrupt handler and polling tasks. This is problemetic for the
following reasons.
a. Full EH context is required for error handling.
Advanced recovery usually involves resetting, command issuing
and other blocking operations.
b. Simple errors may trigger complex error handling behavior.
For example, when an ABRT error occurs, reporting to upper
layer is sufficent for most cases; however, repeated ABRT
errors for known-to-be-supported commands might indicate too
high transmission speed. In such cases, full EH context is
required to perform error handling.
c. Scattered complex EH is difficult to implement and maintain.
EH logic can be somewhat complex and scattering won't help
implementing and maintaining it. Also, libata low level
drivers are allowed to override callbacks where part of EH
logic may reside making matters worse.
[1-3] Synchronization
A simple & concrete qc synchronization model to make sure that EH
and any other processing don't occur concurrently is needed.
[1-4] Clean mechanism to hand off qc's to EH
For EH to handle errors and timeouts, letting EH deal with and
complete both errored and timed out qc's is good for simplicity
and consistency. To achieve this, we need a mechanism to hand off
a qc to EH.
Currently, libata EH has a similar mechanism to hand off a failed
ATAPI qc to EH. As described in libata EH doc, such qc is
half-completed and used as place holder until EH is kicked in and
handles it.
This half-completion isn't very clean semantically and requires
calling splitted internal completion routines directly. Also, as
such qc's are not explicitly marked as failed, not-very-intuitive
stuff has to be done to avoid spurious interrupts or other events
from messing with it after error has occurred.
[1-5] Separate EH qc
EH needs to issue qc's for recovery. There can be several ways to
allocate EH qc.
a. reserve one extra qc for internal/EH commands
b. reserve one of normal qc's
c. use failed qc
d. complete failed qc first and reuse it
The preferred choice is #a for the following reasons.
- Allowing only one concurrent internal command is okay as long as
proper allocation mechanism is implemented or only one user is
guaranteed.
- EH commands are restricted to non-NCQ commands, so reserving an
extra qc won't break qc to tag mapping.
- #b is impossible for non-NCQ devices because only one qc is
available.
- #c requires dancing with qc's internals. No real nerd likes
dancing.
- It may be necessary to issue commands to determine whether to
finish or retry a qc, so #d is out.
[1-6] SCSI/libata separation
Internal libata EH logic implementation should be free from SCSI
considerations. All glueing work should be localized to EH
frontend and once in the actual error handling EH should only deal
with qc's.
[2] Designs
This section proposes detailed design of several important mechanisms
to help discussion and verification.
[2-1] Handoff of failed qc's
As described above, when normal command processing determines that a
qc has failed, those qc's have to be handed off to EH without being
lost.
A new qc flag ATA_QCFLAG_ERROR is defined to mark qc's which have
failed and ata_qc_error() is defined to be used by command processing
to mark failed qc and schedule EH. ata_qc_error() has to be called
under the same condition as ata_qc_complete() - under host_lock - and
performs the following.
1. First check if the command is already marked with
ATA_QCFLAG_ERROR. If so, this isn't the first error completion
attempt, just return.
2. Mark the qc with ATA_QCFLAG_ERROR.
3. As, currently, SCSI command issuing is not atomic with respect to
SHOST_RECOVERY flag, we need a separate atomic mechanism to plug
command issuing. Per-port flag ATA_FLAG_ERROR is set here to
prevent further command issuing.
4. Corresponding scmd's result code is set to
SAM_STAT_CHECK_CONDITION and qc->scsidone() callback is called
directly. As we haven't filled sense data,
scsi_determine_disposition() will return FAILED and SCSI EH will
be scheduled. Note that as we directly call qc->scsidone(), qc is
left intact.
After above function is complete, the following conditions are true.
a. The qc has ATA_QCFLAG_ERROR set and no further normal qc
processing will happen for the command.
b. No new qc will be issued for the port.
c. EH is scheduled.
d. Corresponding scmd and qc are left alone until EH processes them.
Note that to achieve above behavior, we need to modify other places
too. e.g. ata_qc_complete() needs to be modified to ignore failed
qc's and command issuing part to fail issuing if ATA_FLAG_ERROR is
set.
[2-2] Timed out scmd's and qc's
Because libata keeps separate command list as qc array, there can be
disagreement regarding which commands have timed out between SCSI and
libata. Consider the following scenario.
1. A scmd is issued.
2. Corresponding qc is allocated, initialized & issued.
3. SCSI timeout occurs & EH scheduled.
4. The qc completes successfully. Because timer already has expired,
scsi_done() will return without doing anything.
5. EH starts.
In above case, we have a timed out scmd but the corresponding qc has
already completed and been deallocated, and this is the only case
where a failed or timed out scmd doesn't have its corresponding qc.
Note that if the qc failed in step #4, ata_qc_error() would have been
called, the qc tagged with ATA_QCFLAG_ERROR and EH would take steps in
[2-1].
This can be easily worked around by scanning scmds on shost->eh_cmd_q
and complete scmds which don't have corresponding qc's with success
code. This way, internal libata EH can be insulated from SCSI details
and can only deal with qc's.
qc's which are determined to have timed out are marked with
ATA_QCFLAG_ERROR | ATA_QCFLAG_TIMEDOUT. Note that all above should
happen atomically as we don't wanna race with interrupt handler or
polling tasks.
[2-3] Summary of [2-1] and [2-2]
- All failed qc's will have ATA_QCFLAG_ERROR set.
- All timed out qc's will have ATA_QCFLAG_ERROR and
ATA_QCFLAG_TIMEDOUT set.
- Whenever ATA_QCFLAG_EROR bit is set, ATA_FLAG_ERROR should also be
set.
- All of above three should be done while holding host_set lock.
- ata_qc_complete() and ata_qc_error() should not perform any
operation on qc's which have ATA_QCFLAG_ERROR set.
- No non-internal commands should be allowed on ports which have
ATA_FLAG_ERROR set.
[2-4] EH processing & completion
After entered, EH can issue internal qc's for recovery. Note that we
need to implement separate mechanisms for error handling and timeout
as we can't call into EH recursively.
For errors, just reporting failure should be enough and this can be
easily implemented by calling ata_qc_complete() from ata_qc_error()
for internal commands.
Separate timer should be used for internal commands. When this timer
expires, the best we can do is completing the qc with failed status.
EH code should be prepared to take appropriate actions to handle both
errors and timeouts such as resetting device on timeout.
After necessary steps are taken for recovery and disposition is
determined for each failed qc, EH should retry or finish each failed
qc. As noted in SCSI EH doc, eh_strategy_handler() should call either
scsi_finish_command() or scsi_queue_insert(). Because the failed qc's
are still active, overriding their ->scsidone callbacks appropriately
and performing ata_qc_complete() on those will do the job. Note that
ATA_QCFLAG_ERROR checking should be bypassed when finishing off failed
qc's from EH.
After all failed qc's are taken care of, libata EH should make sure
that all integrity constraints described in SCSH EH doc is met and
clear ATA_FLAG_ERROR on the port. Returning from
hostt->eh_strategy_handler() will make SCSI midlayer resume normal
processing.
[3] Ideas
[3-1] Using EH for non-error exceptions and dynamic reconfiguration
Handling non-error exceptions like hot plugging and dynamic
reconfigurations such as transfer speed lowering are best done inside
EH, but currently there is no way to invoke EH without failed or
timedout scmds. IMHO, a mechanism to allow EH invocation without
failed scmd should be simple to implement in SCSI midlayer and can
solve this issue nicely.
[3-2] Using EH for host_set level exclusion
Some EH / configuration actions require host_set level exclusion.
This also can be solved by adding the mechanism described in [3-1].
Before starting such an operation, EH's can be invoked on all other
ports. After all ports are safely parked inside EH, the operation can
be performed. After the operation is complete, other ports can be
released from EH.
[4] Implementation plan
Implementation of new EH can be separated into two stages. The first
is implementing EH framework. i.e. qc handoff, EH invocation, qc
completion in EH and resuming normal operation. The latter part is
implementing actual error handling logic according to ATA exceptions
doc.
After completing the first step, the current error handling logic can
be moved onto the new framework. As this won't change libata's
behavior viewed from controllers and devices, we only have to verify
the framework itself and can continue to use the current logic until
the second part is complete.
As getting error handling logic right would take some time for
testing if not for development and there are some high-on-wishlist
features delayed due to EH - NCQ and hot plugging. Once the new EH
framework is complete, fitting those in first and implementing unified
EH logic later might be a good idea. NCQ can be easily integrated
once the framework is in place, but I'm not sure about hotplugging.
-
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
