On 8/31/10 3:53 PM, Vasu Dev wrote: > Currently fc_queuecommand drops this lock very early on and then re-acquires
Very early but not right away. See below. > this lock just before return, this re-acquired lock gets dropped immediately > by its fast path caller scsi_dispatch_cmd, this re-acquire and then immediate > drop on each IO hits performance especially with small size IOs on multi-core > systems, this hit is significant about 25% with 512 bytes size IOs. > > This lock is not needed in fc_queuecommand and calling fc_queuecommand > without this lock held removes performance hit due to above described > re-acquire and immediately dropping this lock on each IO. > > So this patch adds unlocked_qcmds flag to drop host_lock before > calling only fc_queuecommand and no need to re-acquire and then drop > this lock on each IO, this added flag drops this lock only if LLD wants > as fcoe.ko does here, thus this change won't affect existing LLD since > added unlocked_qcmds flag will be zero in those cases and their host > lock uses would remain same after this patch. > > I considered having this lock dropped inside fc_queuecommand using irq flags > saved by its caller then return without re-acquiring this lock but that > would make this lock usage asymmetric and passing saved irq flags was > another issue with this approach, so RFC to seek any better possible fix > for this or any issue with added unlocked_qcmds while not having > fc_queuecommand under host_lock anymore. I'd appreciate any comments from > Mike, Joe and folks on open-fcoe first before taking this patch to > linux-scsi directly as suggested by Rob and Chris here. > > I also did cable pull test in middle of IOs with this patch and I don't see > any issue with that after this patch and some more testing is already done > successfully with this patch. > > Signed-off-by: Vasu Dev <[email protected]> > --- > drivers/scsi/fcoe/fcoe.c | 1 + > drivers/scsi/libfc/fc_fcp.c | 14 +++++--------- > drivers/scsi/scsi.c | 8 +++++++- > include/scsi/scsi_host.h | 3 +++ > 4 files changed, 16 insertions(+), 10 deletions(-) > > diff --git a/drivers/scsi/fcoe/fcoe.c b/drivers/scsi/fcoe/fcoe.c > index 844d618..280a4df 100644 > --- a/drivers/scsi/fcoe/fcoe.c > +++ b/drivers/scsi/fcoe/fcoe.c > @@ -706,6 +706,7 @@ static int fcoe_shost_config(struct fc_lport *lport, > struct device *dev) > lport->host->max_id = FCOE_MAX_FCP_TARGET; > lport->host->max_channel = 0; > lport->host->max_cmd_len = FCOE_MAX_CMD_LEN; > + lport->host->unlocked_qcmds = 1; > > if (lport->vport) > lport->host->transportt = fcoe_vport_transport_template; > diff --git a/drivers/scsi/libfc/fc_fcp.c b/drivers/scsi/libfc/fc_fcp.c > index 43866e6..39b6bfa 100644 > --- a/drivers/scsi/libfc/fc_fcp.c > +++ b/drivers/scsi/libfc/fc_fcp.c > @@ -1751,8 +1751,7 @@ static inline int fc_fcp_lport_queue_ready(struct > fc_lport *lport) > * @cmd: The scsi_cmnd to be executed > * @done: The callback function to be called when the scsi_cmnd is complete > * > - * This is the i/o strategy routine, called by the SCSI layer. This routine > - * is called with the host_lock held. > + * This is the i/o strategy routine, called by the SCSI layer. > */ > int fc_queuecommand(struct scsi_cmnd *sc_cmd, void (*done)(struct scsi_cmnd > *)) > { > @@ -1770,9 +1769,8 @@ int fc_queuecommand(struct scsi_cmnd *sc_cmd, void > (*done)(struct scsi_cmnd *)) > if (rval) { > sc_cmd->result = rval; > done(sc_cmd); > - return 0; > + return rc; > } Note this isn't quite the first thing. I seem to recall that to avoid races it's necessary to call fc_remote_port_chkready(rport) before dropping the lock. There was a recent patch to move the lock after that. So, maybe the caller can do the chkready for us? I think it's good to eliminate the extra lock taking, somehow. Maybe we could just never drop that lock in queuecommand()? I'm not sure how hard that is. We'd probably run into lock ordering issues with exchange locks, etc. So maybe that's not possible. > - spin_unlock_irq(lport->host->host_lock); > > if (!*(struct fc_remote_port **)rport->dd_data) { > /* > @@ -1781,7 +1779,7 @@ int fc_queuecommand(struct scsi_cmnd *sc_cmd, void > (*done)(struct scsi_cmnd *)) > */ > sc_cmd->result = DID_IMM_RETRY << 16; > done(sc_cmd); > - goto out; > + return rc; > } > > rpriv = rport->dd_data; > @@ -1790,13 +1788,13 @@ int fc_queuecommand(struct scsi_cmnd *sc_cmd, void > (*done)(struct scsi_cmnd *)) > if (lport->qfull) > fc_fcp_can_queue_ramp_down(lport); > rc = SCSI_MLQUEUE_HOST_BUSY; > - goto out; > + return rc; > } > > fsp = fc_fcp_pkt_alloc(lport, GFP_ATOMIC); > if (fsp == NULL) { > rc = SCSI_MLQUEUE_HOST_BUSY; > - goto out; > + return rc; > } > > /* > @@ -1848,8 +1846,6 @@ int fc_queuecommand(struct scsi_cmnd *sc_cmd, void > (*done)(struct scsi_cmnd *)) > fc_fcp_pkt_release(fsp); > rc = SCSI_MLQUEUE_HOST_BUSY; > } > -out: > - spin_lock_irq(lport->host->host_lock); > return rc; > } > EXPORT_SYMBOL(fc_queuecommand); > diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c > index ad0ed21..e7b9f3c 100644 > --- a/drivers/scsi/scsi.c > +++ b/drivers/scsi/scsi.c > @@ -746,6 +746,9 @@ int scsi_dispatch_cmd(struct scsi_cmnd *cmd) > */ > scsi_cmd_get_serial(host, cmd); > > + if (host->unlocked_qcmds) > + spin_unlock_irqrestore(host->host_lock, flags); > + > if (unlikely(host->shost_state == SHOST_DEL)) { > cmd->result = (DID_NO_CONNECT << 16); > scsi_done(cmd); > @@ -753,7 +756,10 @@ int scsi_dispatch_cmd(struct scsi_cmnd *cmd) > trace_scsi_dispatch_cmd_start(cmd); > rtn = host->hostt->queuecommand(cmd, scsi_done); > } > - spin_unlock_irqrestore(host->host_lock, flags); > + > + if (!host->unlocked_qcmds) > + spin_unlock_irqrestore(host->host_lock, flags); > + > if (rtn) { > trace_scsi_dispatch_cmd_error(cmd, rtn); > if (rtn != SCSI_MLQUEUE_DEVICE_BUSY && > diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h > index b7bdecb..1814c51 100644 > --- a/include/scsi/scsi_host.h > +++ b/include/scsi/scsi_host.h > @@ -636,6 +636,9 @@ struct Scsi_Host { > /* Asynchronous scan in progress */ > unsigned async_scan:1; > > + /* call queuecommand without Scsi_Host lock held */ > + unsigned unlocked_qcmds:1; > + > /* > * Optional work queue to be utilized by the transport > */ > > _______________________________________________ > devel mailing list > [email protected] > http://www.open-fcoe.org/mailman/listinfo/devel _______________________________________________ devel mailing list [email protected] http://www.open-fcoe.org/mailman/listinfo/devel
