Konrad Rzeszutek wrote:
>> Ah if your disk are using write back cache then you are going to hit 
>> some problems. So if you see this in /var/log/messages when you loging:
>> kernel: sd 9:0:0:1: [sdb] Write cache: enabled,
>> then later when you run iscsiadm to log out you see:
>> kernel: sd 9:0:0:1: [sdb] Synchronizing SCSI cache
>> Then you are going to hit problems due to the scsi sysfs interface 
>> changing on us. iscsiadm is going to hang. IO is going to hang. You 
>> basically have to reboot the box by hand.
> Mike,
> Are you sure about this? When the sysfs entries are deleted (during the 
> iscsiadm logout phase), the SCSI ml finishes all of the I/Os and the last
> operation is sending the SCSI Cache command. Wouldn't that quiesce I/O ? 
> Granted

See below. You are right if everything goes ok.

> this means doing these steps which are outside the normal iscsiadm:
>  1). flush dirty pages (call 'sync')
>  2). delete the sysfs entries (echo 1 > /sys/block/sdX/device/delete)
>  3). wait until /sys/class/scsi_host/hostZ/host_busy reaches zero
>  4). iscsiadm -m logout

The problem that I described occurs when we run the iscsiadm logout 
command and we used the sysfs delete file. When iscsiadm wrote to that 
attr in 2.6.18 it would return when the cache sync was sent and the 
device was fully deleted in the kernel. In 2.6.21 and above it returned 
right away. So iscsiadm's logout code would write to that attr and think 
the devices were cleaned up, then it would call the iscsi shutdown code 
which would send a logout and cleanup the kernel session, connection and 
host structs, thinking that the devices were properly flushed but IO 
could still be waiting to get written so all types of fun things can 
happen like....

We could get to the scsi host remove call and all the scsi device sysfs 
delete calls would still be starting up, so the host remove call and 
those delete calls would race (so this is we would have bypassed the 
host_busy check in the connection deletion function in the kernel). When 
this happens if the sysfs delete device got the scan mutex first, but 
the iscsi shutdown code had blocked the devices, while we were trying to 
remove the host then the iscsiadm logout command will hang, because the 
delete device would wait forever to try and send the command (it is not 
yet in the host so the command timer is not running and the device is 
blocked), and the remove host call is waiting on the scan mutex which 
the device has.

If you have multiple devices then the remove host command can also end 
up failing IO, because we will have sent the logout and later set the 
session internal state to terminiate and incoming IO on the other 
devices that was queued will be failed when the remove devices functions 
flush the IO.

If you do not have a write back cache we have other problems, where IO 
can be failed when it should not have for the reason above where the 
logout is sent, the terminate bit is set, and the remove host runs 
before the devices were properly removed and that causes IO to be failed.

And actually in some kernels you can hang (the app would hang not 
iscsiadm in this case) when a cache sync was not needed, because if a 
cache sync was not needed when we would remove the host and it would 
delete the device but IO would be stuck in the queues and no one did a 
unplug of the queue when the scsi device was removed. We added a 
iscsi_unblock_session in the iscsi_session_teardown to flush everything 
so at least apps would not hang there (but that resulted in IO getting 
failed like above).

You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/open-iscsi

Reply via email to