Re: Flush outstanding R2Ts during LU Reset
On 08/05/2009 09:53 AM, Hannes Reinecke wrote: > Hannes Reinecke wrote: >> Hannes Reinecke wrote: >>> Hi Mike, >>> >>> finally I've managed to track down the lost TTT issue. >>> >>> We're continuing to get responses from the target >>> even after we have send the TMF PDU (no surprise there). >>> However, the target considers all R2Ts send _before_ >>> the tmf response is sent as valid, requiring us >>> to finish off the R2T sequence by sending a Data-Out >>> PDU even _after_ we have received the TMF response. The problem for this scenario is that I do not think the target is following the RFC, and so we what we do on purpose. We just do not want to oops. We do not care about sending data or completing tasks. 10.6.2. for the initiator has: Receives the task set management response, thus concluding all the tasks in the affected task set. To me, that meant that the task is done. No responses from the target and the initiator should not be sending anything (for iSCSI RFC FastAbort=No all the r2ts should be answered so there should not be anything for use to send). So what happens next for msa and this scenario is a complete mystery to me. What happens if we get tmf response and send data-out/s for the outstanding r2t? 1. Will the target send us more r2ts to complete the full scsi command data transfer for the WRITE? 2. Will it do #1 and when the transfer is done send a scsi response pdu? Will this scsi response pdu indicate success or failure? What am I supposed to do with it? If we leave fail_scsi_task as is then its call in the eh code ended up calling scsi_done then the scsi_cmnd is already back in the mid layer. We can add code to detect this and not, complete the task/cmd in fail_scsi_task. But what about other targets? Do we add other hacks for them? 3. What happens if this is a tape command, and the lu reset inidcates the task did not complete, but we end up sending data for the r2t and that moves the tape and completes the command? My opinion here is us dropping the session first when we detect r2ts still sitting around would better than what we do today, because we are out of the the RFC and I have no idea what is going on. I have no idea what we are supposed to do wrt the RFC and scsi specs and people's data, because I have no idea what the target is going to do next. If we are certain this is the problem, I think it would be best to just bug the msa guys and get it fixed in the target. >>> >>> So I've implemented a (rather hackish) 'flush data out' >>> function, which flushes all outstanding Data out >>> PDUs for a given LUN. Using this things are much >>> better to the point that the 'dropped R2T' messages >>> disappeared. >>> >> Well, I've been to fast. I'm having a rather unfavourable >> (non-)interaction between the xmit thread and the recv >> side; as the recv side runs effectively asynchronous >> to the xmit thread there is a change that wealready >> received the R2T PDU but haven't had a chance to >> process it as it's waiting on the session spinlock. >> Maybe it's an idea to have a per-task spinlock to >> ease the lock contention on the per-session spinlock. I have a patch for that, because after I saw it in fc_fcp.c I thought it would be helpful. It turns out that I am not seeing any performance gain. I think I sent it to the list with some other patches for testing. I will dig it up. >> > And, incidentally, what is the reason for > first setting the SUSPEND bit in iscsi_suspend_tx > and _then_ flush the workqueue? The reason for the suspend call was due to a bug, and could probably be removed, because it was fixed in a different wayfor some other reason. Remember that bug you had where the eh thread was cleaning up a task, but the xmit thread was sending a data-out? I think the original fix was to just suspend the xmit thread by making sure that the xmit thread was not running anymore (the flush work queue) and make sure if someone tried to wake it while we were freeing the task, then the xmit thread would just short circuit. Later, I added refcounting to the tasks and took a refcount in iscsi_data_xmit, so if the eh thread was trying to fail the task, it would not be freed until iscsi_data_xmit was done with it. However, at that time I did not remove the suspend calls from the eh thread paths. I probably did not do this because I forgot or more likely was too much of wuss and was scared in case it was protecting something I did not remember :) But either way (if I am wrong about the RFC here), the whole suspend call could go I think. > Surely this will just empty the workqueue without > any real work done here (as the xmit thread will > be short-circuited anyway). > As we have to flush in the sense of actually > _transmitting_ outstanding requests this looks > (and, in fact, proves) to be rather detrimental > here. > --~--~-~--~~~---~--~~ You received this message because you are subscribed
Re: Flush outstanding R2Ts during LU Reset
Hannes Reinecke wrote: > Hannes Reinecke wrote: >> Hi Mike, >> >> finally I've managed to track down the lost TTT issue. >> >> We're continuing to get responses from the target >> even after we have send the TMF PDU (no surprise there). >> However, the target considers all R2Ts send _before_ >> the tmf response is sent as valid, requiring us >> to finish off the R2T sequence by sending a Data-Out >> PDU even _after_ we have received the TMF response. >> >> So I've implemented a (rather hackish) 'flush data out' >> function, which flushes all outstanding Data out >> PDUs for a given LUN. Using this things are much >> better to the point that the 'dropped R2T' messages >> disappeared. >> > Well, I've been to fast. I'm having a rather unfavourable > (non-)interaction between the xmit thread and the recv > side; as the recv side runs effectively asynchronous > to the xmit thread there is a change that wealready > received the R2T PDU but haven't had a chance to > process it as it's waiting on the session spinlock. > Maybe it's an idea to have a per-task spinlock to > ease the lock contention on the per-session spinlock. > And, incidentally, what is the reason for first setting the SUSPEND bit in iscsi_suspend_tx and _then_ flush the workqueue? Surely this will just empty the workqueue without any real work done here (as the xmit thread will be short-circuited anyway). As we have to flush in the sense of actually _transmitting_ outstanding requests this looks (and, in fact, proves) to be rather detrimental here. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Flush outstanding R2Ts during LU Reset
Hannes Reinecke wrote: > Hi Mike, > > finally I've managed to track down the lost TTT issue. > > We're continuing to get responses from the target > even after we have send the TMF PDU (no surprise there). > However, the target considers all R2Ts send _before_ > the tmf response is sent as valid, requiring us > to finish off the R2T sequence by sending a Data-Out > PDU even _after_ we have received the TMF response. > > So I've implemented a (rather hackish) 'flush data out' > function, which flushes all outstanding Data out > PDUs for a given LUN. Using this things are much > better to the point that the 'dropped R2T' messages > disappeared. > Well, I've been to fast. I'm having a rather unfavourable (non-)interaction between the xmit thread and the recv side; as the recv side runs effectively asynchronous to the xmit thread there is a change that wealready received the R2T PDU but haven't had a chance to process it as it's waiting on the session spinlock. Maybe it's an idea to have a per-task spinlock to ease the lock contention on the per-session spinlock. And of course the whole this would have been far easier if I had a chance to detect a pending r2t by the time fail_scsi_task() is called; then I could just call the xmit thread until the Data-out pdu is sent. Mike, any idea here? Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Flush outstanding R2Ts during LU Reset
Hi Mike, finally I've managed to track down the lost TTT issue. We're continuing to get responses from the target even after we have send the TMF PDU (no surprise there). However, the target considers all R2Ts send _before_ the tmf response is sent as valid, requiring us to finish off the R2T sequence by sending a Data-Out PDU even _after_ we have received the TMF response. So I've implemented a (rather hackish) 'flush data out' function, which flushes all outstanding Data out PDUs for a given LUN. Using this things are much better to the point that the 'dropped R2T' messages disappeared. I'll be sending the patchset for review. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage h...@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---