On 08/05/2009 09:53 AM, Hannes Reinecke wrote:
> Hannes Reinecke wrote:
>> Hannes Reinecke wrote:
>>> Hi Mike,
>>> finally I've managed to track down the lost TTT issue.
>>> We're continuing to get responses from the target
>>> even after we have send the TMF PDU (no surprise there).
>>> However, the target considers all R2Ts send _before_
>>> the tmf response is sent as valid, requiring us
>>> to finish off the R2T sequence by sending a Data-Out
>>> PDU even _after_ we have received the TMF response.

The problem for this scenario is that I do not think the target is 
following the RFC, and so we what we do on purpose. We just do not want 
to oops. We do not care about sending data or completing tasks.

10.6.2. for the initiator has:

Receives the task set management response, thus concluding all the tasks 
in the affected task set.

To me, that meant that the task is done. No responses from the target 
and the initiator should not be sending anything (for iSCSI RFC 
FastAbort=No all the r2ts should be answered so there should not be 
anything for use to send).

So what happens next for msa and this scenario is a complete mystery to 
me.  What happens if we get tmf response and send data-out/s for the 
outstanding r2t?

1. Will the target send us more r2ts to complete the full scsi command 
data transfer for the WRITE?
2. Will it do #1 and when the transfer is done send a scsi response pdu? 
  Will this scsi response pdu indicate success or failure? What am I 
supposed to do with it? If we leave fail_scsi_task as is then its call 
in the eh code ended up calling scsi_done then the scsi_cmnd is already 
back in the mid layer. We can add code to detect this and not, complete 
the task/cmd in fail_scsi_task. But what about other targets? Do we add 
other hacks for them?

3. What happens if this is a tape command, and the lu reset inidcates 
the task did not complete, but we end up sending data for the r2t and 
that moves the tape and completes the command?

My opinion here is us dropping the session first when we detect r2ts 
still sitting around would better than what we do today, because we are 
out of the the RFC and I have no idea what is going on. I have no idea 
what we are supposed to do wrt the RFC and scsi specs and people's data, 
because I have no idea what the target is going to do next.

If we are certain this is the problem, I think it would be best to just 
bug the msa guys and get it fixed in the target.

>>> So I've implemented a (rather hackish) 'flush data out'
>>> function, which flushes all outstanding Data out
>>> PDUs for a given LUN. Using this things are much
>>> better to the point that the 'dropped R2T' messages
>>> disappeared.
>> Well, I've been to fast. I'm having a rather unfavourable
>> (non-)interaction between the xmit thread and the recv
>> side; as the recv side runs effectively asynchronous
>> to the xmit thread there is a change that wealready
>> received the R2T PDU but haven't had a chance to
>> process it as it's waiting on the session spinlock.
>> Maybe it's an idea to have a per-task spinlock to
>> ease the lock contention on the per-session spinlock.

I have a patch for that, because after I saw it in fc_fcp.c I thought it 
would be helpful. It turns out that I am not seeing any performance 
gain. I think I sent it to the list with some other patches for testing. 
I will dig it up.

> And, incidentally, what is the reason for
> first setting the SUSPEND bit in iscsi_suspend_tx
> and _then_ flush the workqueue?

The reason for the suspend call was due to a bug, and could probably be 
removed, because it was fixed in a different wayfor some other reason.

Remember that bug you had where the eh thread was cleaning up a task, 
but the xmit thread was sending a data-out? I think the original fix was 
to just suspend the xmit thread by making sure that the xmit thread was 
not running anymore (the flush work queue) and make sure if someone 
tried to wake it while we were freeing the task, then the xmit thread 
would just short circuit.

Later, I added refcounting to the tasks and took a refcount in 
iscsi_data_xmit, so if the eh thread was trying to fail the task, it 
would not be freed until iscsi_data_xmit was done with it. However, at 
that time I did not remove the suspend calls from the eh thread paths. I 
probably did not do this because I forgot or more likely was too much of 
wuss and was scared in case it was protecting something I did not 
remember :)

But either way (if I am wrong about the RFC here), the whole suspend 
call could go I think.

> Surely this will just empty the workqueue without
> any real work done here (as the xmit thread will
> be short-circuited anyway).
> As we have to flush in the sense of actually
> _transmitting_ outstanding requests this looks
> (and, in fact, proves) to be rather detrimental
> here.

You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to 
For more options, visit this group at http://groups.google.com/group/open-iscsi

Reply via email to