Konrad Rzeszutek wrote:
> On Thu, Apr 17, 2008 at 06:07:12AM -0400, Konrad Rzeszutek wrote:
>>> It looks like the network is off but the session is still running. We 
>>> eventually get to the kernel shutoff here. Is your init script getting 
>>> run? If not then run it. If you left the session on on purpose then you 
>>> cannot turn the network off because the scsi layer will want to do its 
>>> shutdown when the kernel is stopped.
>> Ah. Thanks for the explanation. The init script was run, but it didn't 
>> logoff of all the sessions (it would selectivly logoff instead of doing
>> all of them).
> After I made sure that 'iscsiadm -m session -U all' was called during shutdown
> a QA engineer here was able to make the 'iscsiadm' hang during this sequence.
> The result was that some of the iSCSI sessions did log-out while some other 
> did not,
> and the machine hanged during the "Synchronizing SCSI cache for disk .."

I am not sure if you meant you are running iscsiadm -m session -U all 
and still hanging and if so where so to make sure you are not hitting 
the cache sync bug let me try to seperate all the shutdown bugs we have 
seen :)

iscsiadm will not return until all sessions you asked to logout are 
stopped and the sync caches are completed, but iscsiadm ignores sessions 
marked as being used for boot.

If iscsiadm is run during iscsi init script shutdown then if the network 
is down we will hang and no session should get logged off.

If the network is up and iscsiadm is hanging then it is probably the 
cache sync iscsiadm hang caused by us using the delete sysfs file and 
expecting it not to return until the cache sync has completed. In this 
case some sessions could get logged out right and others could hang. It 
is a race so it is fun and not always predictable :)

If you are hanging when the kernel shutsdown (when the driver model 
shutdown functions get called which for sd.c case will send the cache 
sync) then if your system is not setup we will hang on the cache sync 
because iscsid is not up to handle the connection error that results 
when the scsi eh fails. We can also hang if the network is up right at 
this time, but we experience a connection error or timeout or the target 
could drop connection for whatever reason it wants to. In this case we 
hit the same problem where we send a connection error to userspace but 
iscsid is not there. In older kernels the scsi command would timeout and 
in newer kernels iscsi_eh_cmd_timed_out will keep resetting the command 

> My thought was that since the logic of the SCSI reset is to call first 
> 'iscsi_eh_abort',
> then 'iscsi_eh_device_reset' and then finally 'iscsi_eh_host_reset' we could 
> in
> iscsi_eh_host_reset check the tmf_state as so:
> diff --git a/kernel/libiscsi.c b/kernel/libiscsi.c
> index f8f4cf9..a5b5b37 100644
> --- a/kernel/libiscsi.c
> +++ b/kernel/libiscsi.c
> @@ -1163,7 +1163,8 @@ failed:
>       wait_event_interruptible(conn->ehwait,
>                                session->state == ISCSI_STATE_TERMINATE ||
>                                session->state == ISCSI_STATE_LOGGED_IN ||
> -                              session->state == ISCSI_STATE_RECOVERY_FAILED);
> +                              session->state == ISCSI_STATE_RECOVERY_FAILED 
> ||
> +                              conn->tmf_state == TMF_TIMEDOUT);
>       if (signal_pending(current))
>               flush_signals(current);
> That does make it possible for the SCSI reset sequence to finish off and the 
> machine
> reboots fine. It does not change the behavior during a normal run of iSCSI 
> (ie, unplugging
> a cable for couple of seconds, minutes, etc).

Are you sure about this? I do not think you tried all the timing 
scenarios or had IO running when testing.

> But I have this gnawing feeling that I missing something here ...

The problem with the patch is that if we return SUCCESS from 
iscsi_eh_host_reset then we tell the scsi layer that the problem is 
solved and to retry IO if it can. If we return FAILED then we tell the 
scsi layer we are kaput and it will fail IO.

For a non shutdown run then we want to wait for replacement_timeout 
seconds for the session to come back. If the connection problem was 
detected after the command timer fired so iscsi_eh_cmd_timed_out cannot 
check the status of the connection then we send TMFs they will timeout 
and with your patch we immediately fail the IO. Or if there is a 
connection error while sending TMFs and this causes one to timeout then 
again with your patch we fail the IO too soon.

Also for the shutdown case, it does not handle when the connection 
problem is detected before the command times out. In this case 
iscsi_eh_cmd_timed_out is going to keep asking scsi-ml to reset the 
command timer.

My original solution was to call iscsi_block_session when 
iscsi_conn_failure/ISCSI_SUSPEND_BIT is run. We can do this in newer 
kernels, but for the compat modules this will not work because the 
driver model code used to use a mutex to loop over devices. The compat 
modules problem is probbably not a major issue since we are concerned 
with upstream first, but you just have to make sure that the unblock and 
blocks from userspace (stop and start conn calls) are in order. There 
was some bugs in the initial patch so it did not make it upstream.

The other solution we were tyring is I thought netlink_broadcast would 
return something like ENOTCONN if iscsid was not there. If this was 
always true we could check if we got this error and assume that if we 
tried to tell userspace about something and it was not up then we could 
fast fail the eh path similar to how you did. I think in this case it 
would be ok to ignore the replacement timeout, because iscsid should 
always be up. If it is not then it is because it segfaulted and we are 
screwed not matter what, the user decided to restart it manually which 
is discouraged because we do not support it, or because it is not 
running during those boot and shutdown windows. And for this last there 
is only one solution which is fail.

You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/open-iscsi

Reply via email to