Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-08 Thread Laurence Oberman 
On Thu, 2018-03-08 at 21:42 +0800, Ming Lei wrote:
> On Wed, Mar 07, 2018 at 09:11:37AM -0500, Laurence Oberman wrote:
> > On Tue, 2018-03-06 at 14:24 -0500, Martin K. Petersen wrote:
> > > Ming,
> > > 
> > > > Given both Don and Laurence have verified that patch 1 and
> > > > patch 2
> > > > does fix IO hang, could you consider to merge the two first?
> > > 
> > > Oh, and I would still need a formal Acked-by: from Don and
> > > Tested-by:
> > > from Laurence.
> > > 
> > > Also, for 4.16/scsi-fixes I would prefer verification to be done
> > > with
> > > just patch 1/8 and none of the subsequent changes in place. Just
> > > to
> > > make
> > > sure we're testing the right thing.
> > > 
> > > Thanks!
> > > 
> > 
> > Hello Martin
> > 
> > I tested just Patch 1/8 from the V3 series.
> > No issues running workload and no issues booting on the DL380G7.
> > Don can you ack this so we can at least get this one in.
> > 
> > Against: 4.16.0-rc4.v31of8+ on an x86_64
> > 
> > Tested-by: Laurence Oberman 
> 
> Hi Laurence,
> 
> Thanks for your test!
> 
> Could you test patch 2 too since you have megaraid_sas controller?
> 
> Looks it is better to split the fix patches from the current
> patchset,
> since these fixes should be for V4.16.
> 
> Thanks
> Ming

Ho Ming I see a V4 now so I am going to wait until you split these and
then I will test both HPSA and megaraid_sas once Kashyap agrees.

When I see a V4 show up with the split will pull and act on it.

Thanks
Laurence

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-08 Thread Bart Van Assche 
On Thu, 2018-03-08 at 09:41 +0100, Hannes Reinecke wrote:
> IE the _entire_ request set is allocated as _one_ array, making it quite
> hard to handle from the lower-level CPU caches.
> Also the 'node' indicator doesn't really help us here, as the requests
> have to be access by all CPUs in the shared tag case.
> 
> Would it be possible move tags->rqs to become a _double_ pointer?
> Then we would have only a shared lookup table, but the requests
> themselves can be allocated per node, depending on the CPU map.
> _And_ it should be easier on the CPU cache ...

That is one possible solution. Another solution is to follow the approach
from sbitmap: allocate a single array that is slightly larger than needed
and use the elements in such a way that no two CPUs use the same cache
line.

Bart.

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-08 Thread Ming Lei 
On Wed, Mar 07, 2018 at 09:11:37AM -0500, Laurence Oberman wrote:
> On Tue, 2018-03-06 at 14:24 -0500, Martin K. Petersen wrote:
> > Ming,
> > 
> > > Given both Don and Laurence have verified that patch 1 and patch 2
> > > does fix IO hang, could you consider to merge the two first?
> > 
> > Oh, and I would still need a formal Acked-by: from Don and Tested-by:
> > from Laurence.
> > 
> > Also, for 4.16/scsi-fixes I would prefer verification to be done with
> > just patch 1/8 and none of the subsequent changes in place. Just to
> > make
> > sure we're testing the right thing.
> > 
> > Thanks!
> > 
> 
> Hello Martin
> 
> I tested just Patch 1/8 from the V3 series.
> No issues running workload and no issues booting on the DL380G7.
> Don can you ack this so we can at least get this one in.
> 
> Against: 4.16.0-rc4.v31of8+ on an x86_64
> 
> Tested-by: Laurence Oberman 

Hi Laurence,

Thanks for your test!

Could you test patch 2 too since you have megaraid_sas controller?

Looks it is better to split the fix patches from the current patchset,
since these fixes should be for V4.16.

Thanks
Ming

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-08 Thread Ming Lei 
On Thu, Mar 08, 2018 at 09:41:16AM +0100, Hannes Reinecke wrote:
> On 03/08/2018 09:15 AM, Ming Lei wrote:
> > On Thu, Mar 08, 2018 at 08:50:35AM +0100, Christoph Hellwig wrote:
> >>> +static void hpsa_setup_reply_map(struct ctlr_info *h)
> >>> +{
> >>> + const struct cpumask *mask;
> >>> + unsigned int queue, cpu;
> >>> +
> >>> + for (queue = 0; queue < h->msix_vectors; queue++) {
> >>> + mask = pci_irq_get_affinity(h->pdev, queue);
> >>> + if (!mask)
> >>> + goto fallback;
> >>> +
> >>> + for_each_cpu(cpu, mask)
> >>> + h->reply_map[cpu] = queue;
> >>> + }
> >>> + return;
> >>> +
> >>> +fallback:
> >>> + for_each_possible_cpu(cpu)
> >>> + h->reply_map[cpu] = 0;
> >>> +}
> >>
> >> It seems a little annoying that we have to duplicate this in the driver.
> >> Wouldn't this be solved by your force_blk_mq flag and relying on the
> >> hw_ctx id?
> > 
> > This issue can be solved by force_blk_mq, but may cause performance
> > regression for host-wide tagset drivers:
> > 
> > - If the whole tagset is partitioned into each hw queue, each hw queue's
> > depth may not be high enough, especially SCSI's IO path may be not
> > efficient enough. Even though we keep each queue's depth as 256, which
> > should be high enough to exploit parallelism from device internal view,
> > but still can't get good performance.
> > 
> > - If the whole tagset is still shared among all hw queues, the shared
> > tags can be accessed from all CPUs, and IOPS is degraded.
> > 
> > Kashyap has tested the above two approaches, both hurts IOPS on 
> > megaraid_sas.
> > 
> This is precisely the issue I have been worried about, too.
> 
> The problem is not so much the tagspace (which actually is quite small
> memory footprint-wise), but rather the _requests_ indexed by the tags.

But V1 is done in this way, one shared tags is used and requests are
allocated for each hw queue in NUMA locality, finally Kashyap confirmed
that IOPS can be recovered to normal if iostats is set as 0 after V1 is
applied:

https://marc.info/?l=linux-scsi=151815231026789=2

That means the shared tags does have a big effect on performance.

> 
> We have this:
> 
> struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set,
> unsigned int hctx_idx,
> unsigned int nr_tags,
> unsigned int reserved_tags)
> {
> struct blk_mq_tags *tags;
> int node;
> 
> node = blk_mq_hw_queue_to_node(set->mq_map, hctx_idx);
> if (node == NUMA_NO_NODE)
> node = set->numa_node;
> 
> tags = blk_mq_init_tags(nr_tags, reserved_tags, node,
>  BLK_MQ_FLAG_TO_ALLOC_POLICY(set->flags));
> if (!tags)
> return NULL;
> 
> tags->rqs = kzalloc_node(nr_tags * sizeof(struct request *),
>   GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node);
> 
> 
> IE the _entire_ request set is allocated as _one_ array, making it quite
> hard to handle from the lower-level CPU caches.
> Also the 'node' indicator doesn't really help us here, as the requests
> have to be access by all CPUs in the shared tag case.
> 
> Would it be possible move tags->rqs to become a _double_ pointer?
> Then we would have only a shared lookup table, but the requests
> themselves can be allocated per node, depending on the CPU map.
> _And_ it should be easier on the CPU cache ...

That is basically same with the way in V1, even similar with V3, in
which per-node hw queue is introduced, from Kashyap's test, the
performance isn't bad. I believe finally IOPS can be improved if
scsi_host->host_busy operation is removed from IO path and
megaraid_sas driver is improved, as I mentioned earlier.

Thanks,
Ming

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-08 Thread Hannes Reinecke 
On 03/08/2018 09:15 AM, Ming Lei wrote:
> On Thu, Mar 08, 2018 at 08:50:35AM +0100, Christoph Hellwig wrote:
>>> +static void hpsa_setup_reply_map(struct ctlr_info *h)
>>> +{
>>> +   const struct cpumask *mask;
>>> +   unsigned int queue, cpu;
>>> +
>>> +   for (queue = 0; queue < h->msix_vectors; queue++) {
>>> +   mask = pci_irq_get_affinity(h->pdev, queue);
>>> +   if (!mask)
>>> +   goto fallback;
>>> +
>>> +   for_each_cpu(cpu, mask)
>>> +   h->reply_map[cpu] = queue;
>>> +   }
>>> +   return;
>>> +
>>> +fallback:
>>> +   for_each_possible_cpu(cpu)
>>> +   h->reply_map[cpu] = 0;
>>> +}
>>
>> It seems a little annoying that we have to duplicate this in the driver.
>> Wouldn't this be solved by your force_blk_mq flag and relying on the
>> hw_ctx id?
> 
> This issue can be solved by force_blk_mq, but may cause performance
> regression for host-wide tagset drivers:
> 
> - If the whole tagset is partitioned into each hw queue, each hw queue's
> depth may not be high enough, especially SCSI's IO path may be not
> efficient enough. Even though we keep each queue's depth as 256, which
> should be high enough to exploit parallelism from device internal view,
> but still can't get good performance.
> 
> - If the whole tagset is still shared among all hw queues, the shared
> tags can be accessed from all CPUs, and IOPS is degraded.
> 
> Kashyap has tested the above two approaches, both hurts IOPS on megaraid_sas.
> 
This is precisely the issue I have been worried about, too.

The problem is not so much the tagspace (which actually is quite small
memory footprint-wise), but rather the _requests_ indexed by the tags.

We have this:

struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set,
unsigned int hctx_idx,
unsigned int nr_tags,
unsigned int reserved_tags)
{
struct blk_mq_tags *tags;
int node;

node = blk_mq_hw_queue_to_node(set->mq_map, hctx_idx);
if (node == NUMA_NO_NODE)
node = set->numa_node;

tags = blk_mq_init_tags(nr_tags, reserved_tags, node,
 BLK_MQ_FLAG_TO_ALLOC_POLICY(set->flags));
if (!tags)
return NULL;

tags->rqs = kzalloc_node(nr_tags * sizeof(struct request *),
  GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY, node);


IE the _entire_ request set is allocated as _one_ array, making it quite
hard to handle from the lower-level CPU caches.
Also the 'node' indicator doesn't really help us here, as the requests
have to be access by all CPUs in the shared tag case.

Would it be possible move tags->rqs to become a _double_ pointer?
Then we would have only a shared lookup table, but the requests
themselves can be allocated per node, depending on the CPU map.
_And_ it should be easier on the CPU cache ...

Cheers,

Hannes
-- 
Dr. Hannes ReineckeTeamlead Storage & Networking
h...@suse.de   +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-08 Thread Ming Lei 
On Thu, Mar 08, 2018 at 08:50:35AM +0100, Christoph Hellwig wrote:
> > +static void hpsa_setup_reply_map(struct ctlr_info *h)
> > +{
> > +   const struct cpumask *mask;
> > +   unsigned int queue, cpu;
> > +
> > +   for (queue = 0; queue < h->msix_vectors; queue++) {
> > +   mask = pci_irq_get_affinity(h->pdev, queue);
> > +   if (!mask)
> > +   goto fallback;
> > +
> > +   for_each_cpu(cpu, mask)
> > +   h->reply_map[cpu] = queue;
> > +   }
> > +   return;
> > +
> > +fallback:
> > +   for_each_possible_cpu(cpu)
> > +   h->reply_map[cpu] = 0;
> > +}
> 
> It seems a little annoying that we have to duplicate this in the driver.
> Wouldn't this be solved by your force_blk_mq flag and relying on the
> hw_ctx id?

This issue can be solved by force_blk_mq, but may cause performance
regression for host-wide tagset drivers:

- If the whole tagset is partitioned into each hw queue, each hw queue's
depth may not be high enough, especially SCSI's IO path may be not
efficient enough. Even though we keep each queue's depth as 256, which
should be high enough to exploit parallelism from device internal view,
but still can't get good performance.

- If the whole tagset is still shared among all hw queues, the shared
tags can be accessed from all CPUs, and IOPS is degraded.

Kashyap has tested the above two approaches, both hurts IOPS on megaraid_sas.


thanks,
Ming

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-07 Thread Christoph Hellwig 
> +static void hpsa_setup_reply_map(struct ctlr_info *h)
> +{
> + const struct cpumask *mask;
> + unsigned int queue, cpu;
> +
> + for (queue = 0; queue < h->msix_vectors; queue++) {
> + mask = pci_irq_get_affinity(h->pdev, queue);
> + if (!mask)
> + goto fallback;
> +
> + for_each_cpu(cpu, mask)
> + h->reply_map[cpu] = queue;
> + }
> + return;
> +
> +fallback:
> + for_each_possible_cpu(cpu)
> + h->reply_map[cpu] = 0;
> +}

It seems a little annoying that we have to duplicate this in the driver.
Wouldn't this be solved by your force_blk_mq flag and relying on the
hw_ctx id?

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-07 Thread Laurence Oberman 
On Tue, 2018-03-06 at 14:24 -0500, Martin K. Petersen wrote:
> Ming,
> 
> > Given both Don and Laurence have verified that patch 1 and patch 2
> > does fix IO hang, could you consider to merge the two first?
> 
> Oh, and I would still need a formal Acked-by: from Don and Tested-by:
> from Laurence.
> 
> Also, for 4.16/scsi-fixes I would prefer verification to be done with
> just patch 1/8 and none of the subsequent changes in place. Just to
> make
> sure we're testing the right thing.
> 
> Thanks!
> 

Hello Martin

I tested just Patch 1/8 from the V3 series.
No issues running workload and no issues booting on the DL380G7.
Don can you ack this so we can at least get this one in.

Against: 4.16.0-rc4.v31of8+ on an x86_64

Tested-by: Laurence Oberman 

Thanks
Laurence

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-06 Thread Martin K. Petersen 

Ming,

> Please consider 2/8 too since it is still a fix.

I still need the driver maintainer to ack the change.

-- 
Martin K. Petersen  Oracle Linux Engineering

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-06 Thread Ming Lei 
On Tue, Mar 06, 2018 at 02:24:25PM -0500, Martin K. Petersen wrote:
> 
> Ming,
> 
> > Given both Don and Laurence have verified that patch 1 and patch 2
> > does fix IO hang, could you consider to merge the two first?
> 
> Oh, and I would still need a formal Acked-by: from Don and Tested-by:
> from Laurence.
> 
> Also, for 4.16/scsi-fixes I would prefer verification to be done with
> just patch 1/8 and none of the subsequent changes in place. Just to make
> sure we're testing the right thing.

Hi Martin,

Please consider 2/8 too since it is still a fix.

Thanks,
Ming

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-06 Thread Martin K. Petersen 

Ming,

> Given both Don and Laurence have verified that patch 1 and patch 2
> does fix IO hang, could you consider to merge the two first?

Oh, and I would still need a formal Acked-by: from Don and Tested-by:
from Laurence.

Also, for 4.16/scsi-fixes I would prefer verification to be done with
just patch 1/8 and none of the subsequent changes in place. Just to make
sure we're testing the right thing.

Thanks!

-- 
Martin K. Petersen  Oracle Linux Engineering

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-06 Thread Martin K. Petersen 

Hi Ming,

> Given both Don and Laurence have verified that patch 1 and patch 2
> does fix IO hang, could you consider to merge the two first?

I'm not going to merge the MR patch until Kashyap acks it.

-- 
Martin K. Petersen  Oracle Linux Engineering

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-05 Thread Mike Snitzer 
On Mon, Mar 05 2018 at  2:23am -0500,
Kashyap Desai <kashyap.de...@broadcom.com> wrote:

> > -Original Message-
> > From: Laurence Oberman [mailto:lober...@redhat.com]
> > Sent: Saturday, March 3, 2018 3:23 AM
> > To: Don Brace; Ming Lei
> > Cc: Jens Axboe; linux-bl...@vger.kernel.org; Christoph Hellwig; Mike
> > Snitzer;
> > linux-scsi@vger.kernel.org; Hannes Reinecke; Arun Easi; Omar Sandoval;
> > Martin K . Petersen; James Bottomley; Christoph Hellwig; Kashyap Desai;
> > Peter
> > Rivera; Meelis Roos
> > Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> >
...
> > Unless Kashyap is not happy we need to consider getting this in to Linus
> > now
> > because we are seeing HPE servers that keep hanging now with the original
> > commit now upstream.
> >
> > Kashyap, are you good with the v3 patchset or still concerned with
> > performance. I was getting pretty good IOPS/sec to individual SSD drives
> > set
> > up as jbod devices on the megaraid_sas.
> 
> Laurence -
> Did you find difference with/without the patch ? What was IOPs number with
> and without patch.
> It is not urgent feature, so I would like to take some time to get BRCM's
> performance team involved and do full analysis of performance run and find
> pros/cons.

Performance doesn't matter if the system cannot even boot (e.g. HPE
servers with hpsa using the latest linus tree).

Have you tried your testbed with just applying the first 2 patches?  Or
do those cause the performance hit and the follow-on patches in the
series attempt to recover from it?

Mike

RE: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-05 Thread Don Brace 
> -Original Message-
> From: Kashyap Desai [mailto:kashyap.de...@broadcom.com]
> Sent: Monday, March 05, 2018 1:24 AM
> To: Laurence Oberman <lober...@redhat.com>; Don Brace
> <don.br...@microsemi.com>; Ming Lei <ming@redhat.com>
> Cc: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org; Christoph
> Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com>; linux-
> s...@vger.kernel.org; Hannes Reinecke <h...@suse.de>; Arun Easi
> <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>; Martin K .
> Petersen <martin.peter...@oracle.com>; James Bottomley
> <james.bottom...@hansenpartnership.com>; Christoph Hellwig <h...@lst.de>;
> Peter Rivera <peter.riv...@broadcom.com>; Meelis Roos <mr...@linux.ee>
> Subject: RE: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> 
> EXTERNAL EMAIL
> 
> 
> > -Original Message-
> > From: Laurence Oberman [mailto:lober...@redhat.com]
> > Sent: Saturday, March 3, 2018 3:23 AM
> > To: Don Brace; Ming Lei
> > Cc: Jens Axboe; linux-bl...@vger.kernel.org; Christoph Hellwig; Mike
> > Snitzer;
> > linux-scsi@vger.kernel.org; Hannes Reinecke; Arun Easi; Omar Sandoval;
> > Martin K . Petersen; James Bottomley; Christoph Hellwig; Kashyap Desai;
> > Peter
> > Rivera; Meelis Roos
> > Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> >
> > On Fri, 2018-03-02 at 15:03 +, Don Brace wrote:
> > > > -Original Message-
> > > > From: Laurence Oberman [mailto:lober...@redhat.com]
> > > > Sent: Friday, March 02, 2018 8:09 AM
> > > > To: Ming Lei <ming@redhat.com>
> > > > Cc: Don Brace <don.br...@microsemi.com>; Jens Axboe <axboe@kernel.d
> > > > k>;
> > > > linux-bl...@vger.kernel.org; Christoph Hellwig <h...@infradead.org>;
> > > > Mike Snitzer <snit...@redhat.com>; linux-scsi@vger.kernel.org;
> > > > Hannes Reinecke <h...@suse.de>; Arun Easi <arun.e...@cavium.com>;
> > > > Omar Sandoval <osan...@fb.com>; Martin K . Petersen
> > > > <martin.peter...@oracle.com>; James Bottomley
> > > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig
> > > > <h...@lst.de>; Kashyap Desai <kashyap.de...@broadcom.com>; Peter
> > > > Rivera <peter.riv...@broadcom.com>; Meelis Roos <mr...@linux.ee>
> > > > Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> > > >
> > > > EXTERNAL EMAIL
> > > >
> > > >
> > > > On Fri, 2018-03-02 at 10:16 +0800, Ming Lei wrote:
> > > > > On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> > > > > > On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > > > > > > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > > > > > > -Original Message-
> > > > > > > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > > > > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > > > > > > To: Jens Axboe <ax...@kernel.dk>; linux-block@vger.kernel
> > > > > > > > > .org ; Christoph Hellwig <h...@infradead.org>; Mike Snitzer
> > > > > > > > > <snitzer@redhat .com
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <hare@sus
> > > > > > > > > e.de
> > > > > > > > > > ;
> > > > > > > > >
> > > > > > > > > Arun Easi
> > > > > > > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>;
> > > > > > > > > Martin K .
> > > > > > > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > > > > > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig
> > > > > > > > > ; Don Brace <don.br...@microsemi.com>;
> > > > > > > > > Kashyap Desai <kashyap.de...@broadcom.com>; Peter Rivera
> > > > > > > > > <peter.rivera@ broa dcom .c
> > > > > > > > > om>;
> > > > > > > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > > > > > > <

RE: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-04 Thread Kashyap Desai 
> -Original Message-
> From: Laurence Oberman [mailto:lober...@redhat.com]
> Sent: Saturday, March 3, 2018 3:23 AM
> To: Don Brace; Ming Lei
> Cc: Jens Axboe; linux-bl...@vger.kernel.org; Christoph Hellwig; Mike
> Snitzer;
> linux-scsi@vger.kernel.org; Hannes Reinecke; Arun Easi; Omar Sandoval;
> Martin K . Petersen; James Bottomley; Christoph Hellwig; Kashyap Desai;
> Peter
> Rivera; Meelis Roos
> Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
>
> On Fri, 2018-03-02 at 15:03 +, Don Brace wrote:
> > > -Original Message-
> > > From: Laurence Oberman [mailto:lober...@redhat.com]
> > > Sent: Friday, March 02, 2018 8:09 AM
> > > To: Ming Lei <ming@redhat.com>
> > > Cc: Don Brace <don.br...@microsemi.com>; Jens Axboe <axboe@kernel.d
> > > k>;
> > > linux-bl...@vger.kernel.org; Christoph Hellwig <h...@infradead.org>;
> > > Mike Snitzer <snit...@redhat.com>; linux-scsi@vger.kernel.org;
> > > Hannes Reinecke <h...@suse.de>; Arun Easi <arun.e...@cavium.com>;
> > > Omar Sandoval <osan...@fb.com>; Martin K . Petersen
> > > <martin.peter...@oracle.com>; James Bottomley
> > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig
> > > <h...@lst.de>; Kashyap Desai <kashyap.de...@broadcom.com>; Peter
> > > Rivera <peter.riv...@broadcom.com>; Meelis Roos <mr...@linux.ee>
> > > Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> > >
> > > EXTERNAL EMAIL
> > >
> > >
> > > On Fri, 2018-03-02 at 10:16 +0800, Ming Lei wrote:
> > > > On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> > > > > On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > > > > > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > > > > > -Original Message-
> > > > > > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > > > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > > > > > To: Jens Axboe <ax...@kernel.dk>; linux-block@vger.kernel
> > > > > > > > .org ; Christoph Hellwig <h...@infradead.org>; Mike Snitzer
> > > > > > > > <snitzer@redhat .com
> > > > > > > > >
> > > > > > > >
> > > > > > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <hare@sus
> > > > > > > > e.de
> > > > > > > > > ;
> > > > > > > >
> > > > > > > > Arun Easi
> > > > > > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>;
> > > > > > > > Martin K .
> > > > > > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > > > > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig
> > > > > > > > ; Don Brace <don.br...@microsemi.com>;
> > > > > > > > Kashyap Desai <kashyap.de...@broadcom.com>; Peter Rivera
> > > > > > > > <peter.rivera@ broa dcom .c
> > > > > > > > om>;
> > > > > > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > > > > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > > > > > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply
> > > > > > > > queue
> > > > > > > >
> > > >
> > > > Seems Don run into IO failure without blk-mq, could you run your
> > > > tests again in legacy mode?
> > > >
> > > > Thanks,
> > > > Ming
> > >
> > > Hello Ming
> > > I ran multiple passes on Legacy and still see no issues in my test
> > > bed
> > >
> > > BOOT_IMAGE=/vmlinuz-4.16.0-rc2.ming+ root=UUID=43f86d71-b1bf-4789-
> > > a28e-
> > > 21c6ddc90195 ro crashkernel=256M@64M log_buf_len=64M
> > > console=ttyS1,115200n8
> > >
> > > HEAD of the git kernel I am using
> > >
> > > 694e16f scsi: megaraid: improve scsi_mq performance via .host_tagset
> > > 793686c scsi: hpsa: improve scsi_mq performance via .host_tagset
> > > 60d5b36 block: null_blk: introduce module parameter of 'g_host_tags'
> > > 8847067 scsi: Add template flag 'host_tagset'
> > > a8fbdd6 blk-mq:

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-04 Thread Ming Lei 
On Fri, Mar 02, 2018 at 04:53:21PM -0500, Laurence Oberman wrote:
> On Fri, 2018-03-02 at 15:03 +, Don Brace wrote:
> > > -Original Message-
> > > From: Laurence Oberman [mailto:lober...@redhat.com]
> > > Sent: Friday, March 02, 2018 8:09 AM
> > > To: Ming Lei <ming@redhat.com>
> > > Cc: Don Brace <don.br...@microsemi.com>; Jens Axboe <axboe@kernel.d
> > > k>;
> > > linux-bl...@vger.kernel.org; Christoph Hellwig <h...@infradead.org>;
> > > Mike
> > > Snitzer <snit...@redhat.com>; linux-scsi@vger.kernel.org; Hannes
> > > Reinecke
> > > <h...@suse.de>; Arun Easi <arun.e...@cavium.com>; Omar Sandoval
> > > <osan...@fb.com>; Martin K . Petersen <martin.peter...@oracle.com>;
> > > James
> > > Bottomley <james.bottom...@hansenpartnership.com>; Christoph
> > > Hellwig
> > > <h...@lst.de>; Kashyap Desai <kashyap.de...@broadcom.com>; Peter
> > > Rivera
> > > <peter.riv...@broadcom.com>; Meelis Roos <mr...@linux.ee>
> > > Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply
> > > queue
> > > 
> > > EXTERNAL EMAIL
> > > 
> > > 
> > > On Fri, 2018-03-02 at 10:16 +0800, Ming Lei wrote:
> > > > On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> > > > > On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > > > > > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > > > > > -Original Message-
> > > > > > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > > > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > > > > > To: Jens Axboe <ax...@kernel.dk>; linux-block@vger.kernel
> > > > > > > > .org
> > > > > > > > ;
> > > > > > > > Christoph
> > > > > > > > Hellwig <h...@infradead.org>; Mike Snitzer <snitzer@redhat
> > > > > > > > .com
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <hare@sus
> > > > > > > > e.de
> > > > > > > > > ;
> > > > > > > > 
> > > > > > > > Arun Easi
> > > > > > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>;
> > > > > > > > Martin K
> > > > > > > > .
> > > > > > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > > > > > <james.bottom...@hansenpartnership.com>; Christoph
> > > > > > > > Hellwig  > > > > > > > ch@l
> > > > > > > > st
> > > > > > > > .de>;
> > > > > > > > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > > > > > > > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@
> > > > > > > > broa
> > > > > > > > dcom
> > > > > > > > .c
> > > > > > > > om>;
> > > > > > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > > > > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > > > > > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of
> > > > > > > > reply
> > > > > > > > queue
> > > > > > > > 
> > > > 
> > > > Seems Don run into IO failure without blk-mq, could you run your
> > > > tests again
> > > > in legacy mode?
> > > > 
> > > > Thanks,
> > > > Ming
> > > 
> > > Hello Ming
> > > I ran multiple passes on Legacy and still see no issues in my test
> > > bed
> > > 
> > > BOOT_IMAGE=/vmlinuz-4.16.0-rc2.ming+ root=UUID=43f86d71-b1bf-4789-
> > > a28e-
> > > 21c6ddc90195 ro crashkernel=256M@64M log_buf_len=64M
> > > console=ttyS1,115200n8
> > > 
> > > HEAD of the git kernel I am using
> > > 
> > > 694e16f scsi: megaraid: improve scsi_mq performance via
> > > .host_tagset
> > > 793686c scsi: hpsa: improve scsi_mq performance via .host_tagset
> > > 60d5b36 block: null_blk: introduce module parameter of
> > > 'g_host_tags'
> > > 8847067 scsi: Add template flag 'host_tagset'
> > > a8fbdd6 blk-mq: introduce BLK_MQ_F_HOST_TAGS
> > > 4710fab blk-mq: introduce 'start_tag' field to 'struct blk_mq_tags'
> > > 09bb153 scsi: megaraid_sas: fix selection of reply queue
> > > 52700d8 scsi: hpsa: fix selection of reply queue
> > 
> > I checkout out Linus's tree (4.16.0-rc3+) and re-applied the above
> > patches.
> > I  and have been running 24 hours with no issues.
> > Evidently my forked copy was corrupted. 
> > 
> > So, my I/O testing has gone well. 
> > 
> > I'll run some performance numbers next.
> > 
> > Thanks,
> > Don
> 
> Unless Kashyap is not happy we need to consider getting this in to
> Linus now because we are seeing HPE servers that keep hanging now with
> the original commit now upstream.

Hi Martin,

Given both Don and Laurence have verified that patch 1 and patch 2
does fix IO hang, could you consider to merge the two first?

Thanks,
Ming

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-02 Thread Laurence Oberman 
On Fri, 2018-03-02 at 15:03 +, Don Brace wrote:
> > -Original Message-
> > From: Laurence Oberman [mailto:lober...@redhat.com]
> > Sent: Friday, March 02, 2018 8:09 AM
> > To: Ming Lei <ming@redhat.com>
> > Cc: Don Brace <don.br...@microsemi.com>; Jens Axboe <axboe@kernel.d
> > k>;
> > linux-bl...@vger.kernel.org; Christoph Hellwig <h...@infradead.org>;
> > Mike
> > Snitzer <snit...@redhat.com>; linux-scsi@vger.kernel.org; Hannes
> > Reinecke
> > <h...@suse.de>; Arun Easi <arun.e...@cavium.com>; Omar Sandoval
> > <osan...@fb.com>; Martin K . Petersen <martin.peter...@oracle.com>;
> > James
> > Bottomley <james.bottom...@hansenpartnership.com>; Christoph
> > Hellwig
> > <h...@lst.de>; Kashyap Desai <kashyap.de...@broadcom.com>; Peter
> > Rivera
> > <peter.riv...@broadcom.com>; Meelis Roos <mr...@linux.ee>
> > Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply
> > queue
> > 
> > EXTERNAL EMAIL
> > 
> > 
> > On Fri, 2018-03-02 at 10:16 +0800, Ming Lei wrote:
> > > On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> > > > On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > > > > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > > > > -Original Message-
> > > > > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > > > > To: Jens Axboe <ax...@kernel.dk>; linux-block@vger.kernel
> > > > > > > .org
> > > > > > > ;
> > > > > > > Christoph
> > > > > > > Hellwig <h...@infradead.org>; Mike Snitzer <snitzer@redhat
> > > > > > > .com
> > > > > > > > 
> > > > > > > 
> > > > > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <hare@sus
> > > > > > > e.de
> > > > > > > > ;
> > > > > > > 
> > > > > > > Arun Easi
> > > > > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>;
> > > > > > > Martin K
> > > > > > > .
> > > > > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > > > > <james.bottom...@hansenpartnership.com>; Christoph
> > > > > > > Hellwig  > > > > > > ch@l
> > > > > > > st
> > > > > > > .de>;
> > > > > > > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > > > > > > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@
> > > > > > > broa
> > > > > > > dcom
> > > > > > > .c
> > > > > > > om>;
> > > > > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > > > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > > > > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of
> > > > > > > reply
> > > > > > > queue
> > > > > > > 
> > > 
> > > Seems Don run into IO failure without blk-mq, could you run your
> > > tests again
> > > in legacy mode?
> > > 
> > > Thanks,
> > > Ming
> > 
> > Hello Ming
> > I ran multiple passes on Legacy and still see no issues in my test
> > bed
> > 
> > BOOT_IMAGE=/vmlinuz-4.16.0-rc2.ming+ root=UUID=43f86d71-b1bf-4789-
> > a28e-
> > 21c6ddc90195 ro crashkernel=256M@64M log_buf_len=64M
> > console=ttyS1,115200n8
> > 
> > HEAD of the git kernel I am using
> > 
> > 694e16f scsi: megaraid: improve scsi_mq performance via
> > .host_tagset
> > 793686c scsi: hpsa: improve scsi_mq performance via .host_tagset
> > 60d5b36 block: null_blk: introduce module parameter of
> > 'g_host_tags'
> > 8847067 scsi: Add template flag 'host_tagset'
> > a8fbdd6 blk-mq: introduce BLK_MQ_F_HOST_TAGS
> > 4710fab blk-mq: introduce 'start_tag' field to 'struct blk_mq_tags'
> > 09bb153 scsi: megaraid_sas: fix selection of reply queue
> > 52700d8 scsi: hpsa: fix selection of reply queue
> 
> I checkout out Linus's tree (4.16.0-rc3+) and re-applied the above
> patches.
> I  and have been running 24 hours with no issues.
> Evidently my forked copy was corrupted. 
> 
> So, my I/O testing has gone well. 
> 
> I'll run some performance numbers next.
> 
> Thanks,
> Don

Unless Kashyap is not happy we need to consider getting this in to
Linus now because we are seeing HPE servers that keep hanging now with
the original commit now upstream.

Kashyap, are you good with the v3 patchset or still concerned with
performance. I was getting pretty good IOPS/sec to individual SSD
drives set up as jbod devices on the megaraid_sas.

With larger I/O sizes like 1MB I was getting good MB/sec and not seeing
a measurable performance impact.

I dont have the hardware you have to mimic your configuration.

Thanks
Laurence

RE: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-02 Thread Don Brace 
> -Original Message-
> From: Laurence Oberman [mailto:lober...@redhat.com]
> Sent: Friday, March 02, 2018 8:09 AM
> To: Ming Lei <ming@redhat.com>
> Cc: Don Brace <don.br...@microsemi.com>; Jens Axboe <ax...@kernel.dk>;
> linux-bl...@vger.kernel.org; Christoph Hellwig <h...@infradead.org>; Mike
> Snitzer <snit...@redhat.com>; linux-scsi@vger.kernel.org; Hannes Reinecke
> <h...@suse.de>; Arun Easi <arun.e...@cavium.com>; Omar Sandoval
> <osan...@fb.com>; Martin K . Petersen <martin.peter...@oracle.com>; James
> Bottomley <james.bottom...@hansenpartnership.com>; Christoph Hellwig
> <h...@lst.de>; Kashyap Desai <kashyap.de...@broadcom.com>; Peter Rivera
> <peter.riv...@broadcom.com>; Meelis Roos <mr...@linux.ee>
> Subject: Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> 
> EXTERNAL EMAIL
> 
> 
> On Fri, 2018-03-02 at 10:16 +0800, Ming Lei wrote:
> > On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> > > On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > > > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > > > -Original Message-
> > > > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > > > To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org
> > > > > > ;
> > > > > > Christoph
> > > > > > Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com
> > > > > > >
> > > > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de
> > > > > > >;
> > > > > > Arun Easi
> > > > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>;
> > > > > > Martin K
> > > > > > .
> > > > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig  > > > > > ch@l
> > > > > > st
> > > > > > .de>;
> > > > > > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > > > > > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@broa
> > > > > > dcom
> > > > > > .c
> > > > > > om>;
> > > > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > > > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply
> > > > > > queue
> > > > > >
> > Seems Don run into IO failure without blk-mq, could you run your
> > tests again
> > in legacy mode?
> >
> > Thanks,
> > Ming
> 
> Hello Ming
> I ran multiple passes on Legacy and still see no issues in my test bed
> 
> BOOT_IMAGE=/vmlinuz-4.16.0-rc2.ming+ root=UUID=43f86d71-b1bf-4789-
> a28e-
> 21c6ddc90195 ro crashkernel=256M@64M log_buf_len=64M
> console=ttyS1,115200n8
> 
> HEAD of the git kernel I am using
> 
> 694e16f scsi: megaraid: improve scsi_mq performance via .host_tagset
> 793686c scsi: hpsa: improve scsi_mq performance via .host_tagset
> 60d5b36 block: null_blk: introduce module parameter of 'g_host_tags'
> 8847067 scsi: Add template flag 'host_tagset'
> a8fbdd6 blk-mq: introduce BLK_MQ_F_HOST_TAGS
> 4710fab blk-mq: introduce 'start_tag' field to 'struct blk_mq_tags'
> 09bb153 scsi: megaraid_sas: fix selection of reply queue
> 52700d8 scsi: hpsa: fix selection of reply queue

I checkout out Linus's tree (4.16.0-rc3+) and re-applied the above patches.
I  and have been running 24 hours with no issues.
Evidently my forked copy was corrupted. 

So, my I/O testing has gone well. 

I'll run some performance numbers next.

Thanks,
Don

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-02 Thread Laurence Oberman 
On Fri, 2018-03-02 at 10:16 +0800, Ming Lei wrote:
> On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> > On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > > -Original Message-
> > > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > > To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org
> > > > > ;
> > > > > Christoph
> > > > > Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com
> > > > > >
> > > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de
> > > > > >;
> > > > > Arun Easi
> > > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>;
> > > > > Martin K
> > > > > .
> > > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig  > > > > ch@l
> > > > > st
> > > > > .de>;
> > > > > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > > > > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@broa
> > > > > dcom
> > > > > .c
> > > > > om>;
> > > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply
> > > > > queue
> > > > > 
> > > > > EXTERNAL EMAIL
> > > > > 
> > > > > 
> > > > > From 84676c1f21 (genirq/affinity: assign vectors to all
> > > > > possible
> > > > > CPUs),
> > > > > one msix vector can be created without any online CPU mapped,
> > > > > then
> > > > > one
> > > > > command's completion may not be notified.
> > > > > 
> > > > > This patch setups mapping between cpu and reply queue
> > > > > according
> > > > > to
> > > > > irq
> > > > > affinity info retrived by pci_irq_get_affinity(), and uses
> > > > > this
> > > > > mapping
> > > > > table to choose reply queue for queuing one command.
> > > > > 
> > > > > Then the chosen reply queue has to be active, and fixes IO
> > > > > hang
> > > > > caused
> > > > > by using inactive reply queue which doesn't have any online
> > > > > CPU
> > > > > mapped.
> > > > > 
> > > > > Cc: Hannes Reinecke <h...@suse.de>
> > > > > Cc: Arun Easi <arun.e...@cavium.com>
> > > > > Cc: "Martin K. Petersen" <martin.peter...@oracle.com>,
> > > > > Cc: James Bottomley <james.bottom...@hansenpartnership.com>,
> > > > > Cc: Christoph Hellwig <h...@lst.de>,
> > > > > Cc: Don Brace <don.br...@microsemi.com>
> > > > > Cc: Kashyap Desai <kashyap.de...@broadcom.com>
> > > > > Cc: Peter Rivera <peter.riv...@broadcom.com>
> > > > > Cc: Laurence Oberman <lober...@redhat.com>
> > > > > Cc: Meelis Roos <mr...@linux.ee>
> > > > > Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all
> > > > > possible CPUs")
> > > > > Signed-off-by: Ming Lei <ming@redhat.com>
> > > > 
> > > > I am getting some issues that need to be tracked down:
> > > > 
> > > > [ 1636.032984] hpsa :87:00.0: Acknowledging event:
> > > > 0xc032
> > > > (HP
> > > > SSD Smart Path configuration change)
> > > > [ 1638.510656] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > > > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap-
> > > > En-
> > > > Exp=0
> > > > [ 1653.967695] hpsa :87:00.0: Acknowledging event:
> > > > 0x8020
> > > > (HP
> > > > SSD Smart Path configuration change)
> > > > [ 1656.770377] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > > > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap-
> > > > En-
> > > > Exp=0
>

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-01 Thread Ming Lei 
On Thu, Mar 01, 2018 at 04:19:34PM -0500, Laurence Oberman wrote:
> On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> > On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > > -Original Message-
> > > > From: Ming Lei [mailto:ming@redhat.com]
> > > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > > To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org;
> > > > Christoph
> > > > Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com>
> > > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de>;
> > > > Arun Easi
> > > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>; Martin K
> > > > .
> > > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig <hch@l
> > > > st
> > > > .de>;
> > > > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > > > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@broadcom
> > > > .c
> > > > om>;
> > > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> > > > 
> > > > EXTERNAL EMAIL
> > > > 
> > > > 
> > > > From 84676c1f21 (genirq/affinity: assign vectors to all possible
> > > > CPUs),
> > > > one msix vector can be created without any online CPU mapped,
> > > > then
> > > > one
> > > > command's completion may not be notified.
> > > > 
> > > > This patch setups mapping between cpu and reply queue according
> > > > to
> > > > irq
> > > > affinity info retrived by pci_irq_get_affinity(), and uses this
> > > > mapping
> > > > table to choose reply queue for queuing one command.
> > > > 
> > > > Then the chosen reply queue has to be active, and fixes IO hang
> > > > caused
> > > > by using inactive reply queue which doesn't have any online CPU
> > > > mapped.
> > > > 
> > > > Cc: Hannes Reinecke <h...@suse.de>
> > > > Cc: Arun Easi <arun.e...@cavium.com>
> > > > Cc: "Martin K. Petersen" <martin.peter...@oracle.com>,
> > > > Cc: James Bottomley <james.bottom...@hansenpartnership.com>,
> > > > Cc: Christoph Hellwig <h...@lst.de>,
> > > > Cc: Don Brace <don.br...@microsemi.com>
> > > > Cc: Kashyap Desai <kashyap.de...@broadcom.com>
> > > > Cc: Peter Rivera <peter.riv...@broadcom.com>
> > > > Cc: Laurence Oberman <lober...@redhat.com>
> > > > Cc: Meelis Roos <mr...@linux.ee>
> > > > Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all
> > > > possible CPUs")
> > > > Signed-off-by: Ming Lei <ming@redhat.com>
> > > 
> > > I am getting some issues that need to be tracked down:
> > > 
> > > [ 1636.032984] hpsa :87:00.0: Acknowledging event: 0xc032
> > > (HP
> > > SSD Smart Path configuration change)
> > > [ 1638.510656] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> > > Exp=0
> > > [ 1653.967695] hpsa :87:00.0: Acknowledging event: 0x8020
> > > (HP
> > > SSD Smart Path configuration change)
> > > [ 1656.770377] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> > > Exp=0
> > > [ 2839.762267] hpsa :87:00.0: Acknowledging event: 0x8020
> > > (HP
> > > SSD Smart Path configuration change)
> > > [ 2840.841290] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> > > Exp=0
> > > [ 2917.582653] hpsa :87:00.0: Acknowledging event: 0xc020
> > > (HP
> > > SSD Smart Path configuration change)
> > > [ 2919.087191] hpsa :87:00.0: scsi 3:1:0:1: updated Direct-
> > > Access HP   LOGICAL VOLUME   RAID-5 SSDSmartPathCap+ En+
> > > Exp=1
> > > [ 2919.142527] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> > > [3:1:0:2] A phys disk component of LV is missing

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-01 Thread Ming Lei 
Hi Don,

Thanks for your test!

On Thu, Mar 01, 2018 at 04:18:17PM +, Don Brace wrote:
> > -Original Message-
> > From: Ming Lei [mailto:ming@redhat.com]
> > Sent: Tuesday, February 27, 2018 4:08 AM
> > To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org; Christoph
> > Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com>
> > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de>; Arun Easi
> > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>; Martin K .
> > Petersen <martin.peter...@oracle.com>; James Bottomley
> > <james.bottom...@hansenpartnership.com>; Christoph Hellwig <h...@lst.de>;
> > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > <kashyap.de...@broadcom.com>; Peter Rivera <peter.riv...@broadcom.com>;
> > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> > 
> > EXTERNAL EMAIL
> > 
> > 
> > From 84676c1f21 (genirq/affinity: assign vectors to all possible CPUs),
> > one msix vector can be created without any online CPU mapped, then one
> > command's completion may not be notified.
> > 
> > This patch setups mapping between cpu and reply queue according to irq
> > affinity info retrived by pci_irq_get_affinity(), and uses this mapping
> > table to choose reply queue for queuing one command.
> > 
> > Then the chosen reply queue has to be active, and fixes IO hang caused
> > by using inactive reply queue which doesn't have any online CPU mapped.
> > 
> > Cc: Hannes Reinecke <h...@suse.de>
> > Cc: Arun Easi <arun.e...@cavium.com>
> > Cc: "Martin K. Petersen" <martin.peter...@oracle.com>,
> > Cc: James Bottomley <james.bottom...@hansenpartnership.com>,
> > Cc: Christoph Hellwig <h...@lst.de>,
> > Cc: Don Brace <don.br...@microsemi.com>
> > Cc: Kashyap Desai <kashyap.de...@broadcom.com>
> > Cc: Peter Rivera <peter.riv...@broadcom.com>
> > Cc: Laurence Oberman <lober...@redhat.com>
> > Cc: Meelis Roos <mr...@linux.ee>
> > Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all possible CPUs")
> > Signed-off-by: Ming Lei <ming@redhat.com>
> 
> I am getting some issues that need to be tracked down:

I check the patch one more time, not find odd thing, and the only one
is that inside hpsa_do_reset(), wait_for_device_to_become_ready() is
called to send 'test unit ready' always by the reply queue 0. Do you know
if something bad may happen if other non-zero reply queue is used?

Could you share us how you reproduce this issue?

Looks you can boot successfully, so could you please provide the
following output?

1) what is your server type? We may find one in our lab, so that I can
try to reproduce it.

2) lscpu

3) irq affinity info, and you need to pass the 1st column of
'lspci' of your hpsa PCI device to this script:

#!/bin/sh
if [ $# -ge 1 ]; then
PCID=$1
else
PCID=`lspci | grep "Non-Volatile memory" | cut -c1-7`
fi
PCIP=`find /sys/devices -name *$PCID | grep pci`
IRQS=`ls $PCIP/msi_irqs`

echo "kernel version: "
uname -a

echo "PCI name is $PCID, dump its irq affinity:"
for IRQ in $IRQS; do
CPUS=`cat /proc/irq/$IRQ/smp_affinity_list`
echo "\tirq $IRQ, cpu list $CPUS"
done


Thanks,
Ming

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-01 Thread Laurence Oberman 
On Thu, 2018-03-01 at 14:01 -0500, Laurence Oberman wrote:
> On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > > -Original Message-
> > > From: Ming Lei [mailto:ming@redhat.com]
> > > Sent: Tuesday, February 27, 2018 4:08 AM
> > > To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org;
> > > Christoph
> > > Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com>
> > > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de>;
> > > Arun Easi
> > > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>; Martin K
> > > .
> > > Petersen <martin.peter...@oracle.com>; James Bottomley
> > > <james.bottom...@hansenpartnership.com>; Christoph Hellwig <hch@l
> > > st
> > > .de>;
> > > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@broadcom
> > > .c
> > > om>;
> > > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> > > 
> > > EXTERNAL EMAIL
> > > 
> > > 
> > > From 84676c1f21 (genirq/affinity: assign vectors to all possible
> > > CPUs),
> > > one msix vector can be created without any online CPU mapped,
> > > then
> > > one
> > > command's completion may not be notified.
> > > 
> > > This patch setups mapping between cpu and reply queue according
> > > to
> > > irq
> > > affinity info retrived by pci_irq_get_affinity(), and uses this
> > > mapping
> > > table to choose reply queue for queuing one command.
> > > 
> > > Then the chosen reply queue has to be active, and fixes IO hang
> > > caused
> > > by using inactive reply queue which doesn't have any online CPU
> > > mapped.
> > > 
> > > Cc: Hannes Reinecke <h...@suse.de>
> > > Cc: Arun Easi <arun.e...@cavium.com>
> > > Cc: "Martin K. Petersen" <martin.peter...@oracle.com>,
> > > Cc: James Bottomley <james.bottom...@hansenpartnership.com>,
> > > Cc: Christoph Hellwig <h...@lst.de>,
> > > Cc: Don Brace <don.br...@microsemi.com>
> > > Cc: Kashyap Desai <kashyap.de...@broadcom.com>
> > > Cc: Peter Rivera <peter.riv...@broadcom.com>
> > > Cc: Laurence Oberman <lober...@redhat.com>
> > > Cc: Meelis Roos <mr...@linux.ee>
> > > Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all
> > > possible CPUs")
> > > Signed-off-by: Ming Lei <ming@redhat.com>
> > 
> > I am getting some issues that need to be tracked down:
> > 
> > [ 1636.032984] hpsa :87:00.0: Acknowledging event: 0xc032
> > (HP
> > SSD Smart Path configuration change)
> > [ 1638.510656] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> > Exp=0
> > [ 1653.967695] hpsa :87:00.0: Acknowledging event: 0x8020
> > (HP
> > SSD Smart Path configuration change)
> > [ 1656.770377] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> > Exp=0
> > [ 2839.762267] hpsa :87:00.0: Acknowledging event: 0x8020
> > (HP
> > SSD Smart Path configuration change)
> > [ 2840.841290] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> > Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> > Exp=0
> > [ 2917.582653] hpsa :87:00.0: Acknowledging event: 0xc020
> > (HP
> > SSD Smart Path configuration change)
> > [ 2919.087191] hpsa :87:00.0: scsi 3:1:0:1: updated Direct-
> > Access HP   LOGICAL VOLUME   RAID-5 SSDSmartPathCap+ En+
> > Exp=1
> > [ 2919.142527] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> > [3:1:0:2] A phys disk component of LV is missing, turning off
> > offload_enabled for LV.
> > [ 2919.203915] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> > [3:1:0:2] A phys disk component of LV is missing, turning off
> > offload_enabled for LV.
> > [ 2919.266921] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> > [3:1:0:2] A phys disk component of LV is missing, turning off
> > offload_enabled for LV.
> > [ 2934.999629] hpsa :87:00.0: Acknowledging event: 0x4000
> > (HP
> > SSD

Re: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-01 Thread Laurence Oberman 
On Thu, 2018-03-01 at 16:18 +, Don Brace wrote:
> > -Original Message-
> > From: Ming Lei [mailto:ming@redhat.com]
> > Sent: Tuesday, February 27, 2018 4:08 AM
> > To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org;
> > Christoph
> > Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com>
> > Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de>;
> > Arun Easi
> > <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>; Martin K .
> > Petersen <martin.peter...@oracle.com>; James Bottomley
> > <james.bottom...@hansenpartnership.com>; Christoph Hellwig <hch@lst
> > .de>;
> > Don Brace <don.br...@microsemi.com>; Kashyap Desai
> > <kashyap.de...@broadcom.com>; Peter Rivera <peter.rivera@broadcom.c
> > om>;
> > Laurence Oberman <lober...@redhat.com>; Ming Lei
> > <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> > Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> > 
> > EXTERNAL EMAIL
> > 
> > 
> > From 84676c1f21 (genirq/affinity: assign vectors to all possible
> > CPUs),
> > one msix vector can be created without any online CPU mapped, then
> > one
> > command's completion may not be notified.
> > 
> > This patch setups mapping between cpu and reply queue according to
> > irq
> > affinity info retrived by pci_irq_get_affinity(), and uses this
> > mapping
> > table to choose reply queue for queuing one command.
> > 
> > Then the chosen reply queue has to be active, and fixes IO hang
> > caused
> > by using inactive reply queue which doesn't have any online CPU
> > mapped.
> > 
> > Cc: Hannes Reinecke <h...@suse.de>
> > Cc: Arun Easi <arun.e...@cavium.com>
> > Cc: "Martin K. Petersen" <martin.peter...@oracle.com>,
> > Cc: James Bottomley <james.bottom...@hansenpartnership.com>,
> > Cc: Christoph Hellwig <h...@lst.de>,
> > Cc: Don Brace <don.br...@microsemi.com>
> > Cc: Kashyap Desai <kashyap.de...@broadcom.com>
> > Cc: Peter Rivera <peter.riv...@broadcom.com>
> > Cc: Laurence Oberman <lober...@redhat.com>
> > Cc: Meelis Roos <mr...@linux.ee>
> > Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all
> > possible CPUs")
> > Signed-off-by: Ming Lei <ming@redhat.com>
> 
> I am getting some issues that need to be tracked down:
> 
> [ 1636.032984] hpsa :87:00.0: Acknowledging event: 0xc032 (HP
> SSD Smart Path configuration change)
> [ 1638.510656] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> Exp=0
> [ 1653.967695] hpsa :87:00.0: Acknowledging event: 0x8020 (HP
> SSD Smart Path configuration change)
> [ 1656.770377] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> Exp=0
> [ 2839.762267] hpsa :87:00.0: Acknowledging event: 0x8020 (HP
> SSD Smart Path configuration change)
> [ 2840.841290] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-
> Access HP   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En-
> Exp=0
> [ 2917.582653] hpsa :87:00.0: Acknowledging event: 0xc020 (HP
> SSD Smart Path configuration change)
> [ 2919.087191] hpsa :87:00.0: scsi 3:1:0:1: updated Direct-
> Access HP   LOGICAL VOLUME   RAID-5 SSDSmartPathCap+ En+
> Exp=1
> [ 2919.142527] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> [3:1:0:2] A phys disk component of LV is missing, turning off
> offload_enabled for LV.
> [ 2919.203915] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> [3:1:0:2] A phys disk component of LV is missing, turning off
> offload_enabled for LV.
> [ 2919.266921] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> [3:1:0:2] A phys disk component of LV is missing, turning off
> offload_enabled for LV.
> [ 2934.999629] hpsa :87:00.0: Acknowledging event: 0x4000 (HP
> SSD Smart Path state change)
> [ 2936.937333] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> [3:1:0:2] A phys disk component of LV is missing, turning off
> offload_enabled for LV.
> [ 2936.998707] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> [3:1:0:2] A phys disk component of LV is missing, turning off
> offload_enabled for LV.
> [ 2937.060101] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs:
> [3:1:0:2] A phys disk component of LV is missing, turning off
> offload_enabled for LV.
> [ 3619.711122] sd 3:1:0:3: [sde] tag#436 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE

RE: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-03-01 Thread Don Brace 
> -Original Message-
> From: Ming Lei [mailto:ming@redhat.com]
> Sent: Tuesday, February 27, 2018 4:08 AM
> To: Jens Axboe <ax...@kernel.dk>; linux-bl...@vger.kernel.org; Christoph
> Hellwig <h...@infradead.org>; Mike Snitzer <snit...@redhat.com>
> Cc: linux-scsi@vger.kernel.org; Hannes Reinecke <h...@suse.de>; Arun Easi
> <arun.e...@cavium.com>; Omar Sandoval <osan...@fb.com>; Martin K .
> Petersen <martin.peter...@oracle.com>; James Bottomley
> <james.bottom...@hansenpartnership.com>; Christoph Hellwig <h...@lst.de>;
> Don Brace <don.br...@microsemi.com>; Kashyap Desai
> <kashyap.de...@broadcom.com>; Peter Rivera <peter.riv...@broadcom.com>;
> Laurence Oberman <lober...@redhat.com>; Ming Lei
> <ming@redhat.com>; Meelis Roos <mr...@linux.ee>
> Subject: [PATCH V3 1/8] scsi: hpsa: fix selection of reply queue
> 
> EXTERNAL EMAIL
> 
> 
> From 84676c1f21 (genirq/affinity: assign vectors to all possible CPUs),
> one msix vector can be created without any online CPU mapped, then one
> command's completion may not be notified.
> 
> This patch setups mapping between cpu and reply queue according to irq
> affinity info retrived by pci_irq_get_affinity(), and uses this mapping
> table to choose reply queue for queuing one command.
> 
> Then the chosen reply queue has to be active, and fixes IO hang caused
> by using inactive reply queue which doesn't have any online CPU mapped.
> 
> Cc: Hannes Reinecke <h...@suse.de>
> Cc: Arun Easi <arun.e...@cavium.com>
> Cc: "Martin K. Petersen" <martin.peter...@oracle.com>,
> Cc: James Bottomley <james.bottom...@hansenpartnership.com>,
> Cc: Christoph Hellwig <h...@lst.de>,
> Cc: Don Brace <don.br...@microsemi.com>
> Cc: Kashyap Desai <kashyap.de...@broadcom.com>
> Cc: Peter Rivera <peter.riv...@broadcom.com>
> Cc: Laurence Oberman <lober...@redhat.com>
> Cc: Meelis Roos <mr...@linux.ee>
> Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all possible CPUs")
> Signed-off-by: Ming Lei <ming@redhat.com>

I am getting some issues that need to be tracked down:

[ 1636.032984] hpsa :87:00.0: Acknowledging event: 0xc032 (HP SSD Smart 
Path configuration change)
[ 1638.510656] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-Access HP
   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En- Exp=0
[ 1653.967695] hpsa :87:00.0: Acknowledging event: 0x8020 (HP SSD Smart 
Path configuration change)
[ 1656.770377] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-Access HP
   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En- Exp=0
[ 2839.762267] hpsa :87:00.0: Acknowledging event: 0x8020 (HP SSD Smart 
Path configuration change)
[ 2840.841290] hpsa :87:00.0: scsi 3:0:8:0: updated Direct-Access HP
   MO0400JDVEU  PHYS DRV SSDSmartPathCap- En- Exp=0
[ 2917.582653] hpsa :87:00.0: Acknowledging event: 0xc020 (HP SSD Smart 
Path configuration change)
[ 2919.087191] hpsa :87:00.0: scsi 3:1:0:1: updated Direct-Access HP
   LOGICAL VOLUME   RAID-5 SSDSmartPathCap+ En+ Exp=1
[ 2919.142527] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs: [3:1:0:2] A phys 
disk component of LV is missing, turning off offload_enabled for LV.
[ 2919.203915] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs: [3:1:0:2] A phys 
disk component of LV is missing, turning off offload_enabled for LV.
[ 2919.266921] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs: [3:1:0:2] A phys 
disk component of LV is missing, turning off offload_enabled for LV.
[ 2934.999629] hpsa :87:00.0: Acknowledging event: 0x4000 (HP SSD Smart 
Path state change)
[ 2936.937333] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs: [3:1:0:2] A phys 
disk component of LV is missing, turning off offload_enabled for LV.
[ 2936.998707] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs: [3:1:0:2] A phys 
disk component of LV is missing, turning off offload_enabled for LV.
[ 2937.060101] hpsa :87:00.0: hpsa_figure_phys_disk_ptrs: [3:1:0:2] A phys 
disk component of LV is missing, turning off offload_enabled for LV.
[ 3619.711122] sd 3:1:0:3: [sde] tag#436 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[ 3619.751150] sd 3:1:0:3: [sde] tag#436 Sense Key : Aborted Command [current] 
[ 3619.784375] sd 3:1:0:3: [sde] tag#436 Add. Sense: Internal target failure
[ 3619.816530] sd 3:1:0:3: [sde] tag#436 CDB: Read(10) 28 00 01 1b ad af 00 00 
01 00
[ 3619.852295] print_req_error: I/O error, dev sde, sector 18591151
[ 3619.880850] sd 3:1:0:3: [sde] tag#461 FAILED Result: hostbyte=DID_OK 
driverbyte=DRIVER_SENSE
[ 3619.920981] sd 3:1:0:3: [sde] tag#461 Sense Key : Aborted Command [current] 
[ 3619.955081] sd 3:1:0:3: [sde] tag#461 Add. Sense: Internal target failure
[ 3619.98705

[PATCH V3 1/8] scsi: hpsa: fix selection of reply queue

2018-02-27 Thread Ming Lei 
>From 84676c1f21 (genirq/affinity: assign vectors to all possible CPUs),
one msix vector can be created without any online CPU mapped, then one
command's completion may not be notified.

This patch setups mapping between cpu and reply queue according to irq
affinity info retrived by pci_irq_get_affinity(), and uses this mapping
table to choose reply queue for queuing one command.

Then the chosen reply queue has to be active, and fixes IO hang caused
by using inactive reply queue which doesn't have any online CPU mapped.

Cc: Hannes Reinecke 
Cc: Arun Easi 
Cc: "Martin K. Petersen" ,
Cc: James Bottomley ,
Cc: Christoph Hellwig ,
Cc: Don Brace 
Cc: Kashyap Desai 
Cc: Peter Rivera 
Cc: Laurence Oberman 
Cc: Meelis Roos 
Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all possible CPUs")
Signed-off-by: Ming Lei 
---
 drivers/scsi/hpsa.c | 73 +++--
 drivers/scsi/hpsa.h |  1 +
 2 files changed, 55 insertions(+), 19 deletions(-)

diff --git a/drivers/scsi/hpsa.c b/drivers/scsi/hpsa.c
index 5293e6827ce5..3a9eca163db8 100644
--- a/drivers/scsi/hpsa.c
+++ b/drivers/scsi/hpsa.c
@@ -1045,11 +1045,7 @@ static void set_performant_mode(struct ctlr_info *h, 
struct CommandList *c,
c->busaddr |= 1 | (h->blockFetchTable[c->Header.SGList] << 1);
if (unlikely(!h->msix_vectors))
return;
-   if (likely(reply_queue == DEFAULT_REPLY_QUEUE))
-   c->Header.ReplyQueue =
-   raw_smp_processor_id() % h->nreply_queues;
-   else
-   c->Header.ReplyQueue = reply_queue % h->nreply_queues;
+   c->Header.ReplyQueue = reply_queue;
}
 }
 
@@ -1063,10 +1059,7 @@ static void set_ioaccel1_performant_mode(struct 
ctlr_info *h,
 * Tell the controller to post the reply to the queue for this
 * processor.  This seems to give the best I/O throughput.
 */
-   if (likely(reply_queue == DEFAULT_REPLY_QUEUE))
-   cp->ReplyQueue = smp_processor_id() % h->nreply_queues;
-   else
-   cp->ReplyQueue = reply_queue % h->nreply_queues;
+   cp->ReplyQueue = reply_queue;
/*
 * Set the bits in the address sent down to include:
 *  - performant mode bit (bit 0)
@@ -1087,10 +1080,7 @@ static void set_ioaccel2_tmf_performant_mode(struct 
ctlr_info *h,
/* Tell the controller to post the reply to the queue for this
 * processor.  This seems to give the best I/O throughput.
 */
-   if (likely(reply_queue == DEFAULT_REPLY_QUEUE))
-   cp->reply_queue = smp_processor_id() % h->nreply_queues;
-   else
-   cp->reply_queue = reply_queue % h->nreply_queues;
+   cp->reply_queue = reply_queue;
/* Set the bits in the address sent down to include:
 *  - performant mode bit not used in ioaccel mode 2
 *  - pull count (bits 0-3)
@@ -1109,10 +1099,7 @@ static void set_ioaccel2_performant_mode(struct 
ctlr_info *h,
 * Tell the controller to post the reply to the queue for this
 * processor.  This seems to give the best I/O throughput.
 */
-   if (likely(reply_queue == DEFAULT_REPLY_QUEUE))
-   cp->reply_queue = smp_processor_id() % h->nreply_queues;
-   else
-   cp->reply_queue = reply_queue % h->nreply_queues;
+   cp->reply_queue = reply_queue;
/*
 * Set the bits in the address sent down to include:
 *  - performant mode bit not used in ioaccel mode 2
@@ -1157,6 +1144,8 @@ static void __enqueue_cmd_and_start_io(struct ctlr_info 
*h,
 {
dial_down_lockup_detection_during_fw_flash(h, c);
atomic_inc(>commands_outstanding);
+
+   reply_queue = h->reply_map[raw_smp_processor_id()];
switch (c->cmd_type) {
case CMD_IOACCEL1:
set_ioaccel1_performant_mode(h, c, reply_queue);
@@ -7376,6 +7365,26 @@ static void hpsa_disable_interrupt_mode(struct ctlr_info 
*h)
h->msix_vectors = 0;
 }
 
+static void hpsa_setup_reply_map(struct ctlr_info *h)
+{
+   const struct cpumask *mask;
+   unsigned int queue, cpu;
+
+   for (queue = 0; queue < h->msix_vectors; queue++) {
+   mask = pci_irq_get_affinity(h->pdev, queue);
+   if (!mask)
+   goto fallback;
+
+   for_each_cpu(cpu, mask)
+   h->reply_map[cpu] = queue;
+   }
+   return;
+
+fallback:
+   for_each_possible_cpu(cpu)
+   h->reply_map[cpu] = 0;
+}
+
 /* If MSI/MSI-X is supported by the kernel we will try to enable it on
  * controllers that are capable. If not, we