RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Tuesday, February 13, 2018 6:11 AM > To: Kashyap Desai > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > Hi Kashyap, > > On Tue, Feb 13, 2018 at 12:05:14AM +0530, Kashyap Desai wrote: > > > -Original Message- > > > From: Ming Lei [mailto:ming@redhat.com] > > > Sent: Sunday, February 11, 2018 11:01 AM > > > To: Kashyap Desai > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > Easi; Omar > > Sandoval; > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > Peter > > > Rivera; Paolo Bonzini; Laurence Oberman > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > introduce force_blk_mq > > > > > > On Sat, Feb 10, 2018 at 09:00:57AM +0800, Ming Lei wrote: > > > > Hi Kashyap, > > > > > > > > On Fri, Feb 09, 2018 at 02:12:16PM +0530, Kashyap Desai wrote: > > > > > > -Original Message- > > > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > > > Sent: Friday, February 9, 2018 11:01 AM > > > > > > To: Kashyap Desai > > > > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; > > > > > > Arun Easi; Omar > > > > > Sandoval; > > > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > > > Brace; > > > > > Peter > > > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > > introduce force_blk_mq > > > > > > > > > > > > On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > > > > > > > -Original Message- > > > > > > > > From: Ming Lei [mailto:ming....@redhat.com] > > > > > > > > Sent: Thursday, February 8, 2018 10:23 PM > > > > > > > > To: Hannes Reinecke > > > > > > > > Cc: Kashyap Desai; Jens Axboe; > > > > > > > > linux-bl...@vger.kernel.org; Christoph Hellwig; Mike > > > > > > > > Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > > > > > > > Sandoval; > > > > > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; > > > > > > > > Don Brace; > > > > > > > Peter > > > > > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global > > > > > > > > tags & introduce force_blk_mq > > > > > > > > > > > > > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke > > wrote: > > > > > > > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > > > > > > > >> -Original Message- > > > > > > > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > > > > > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > > > > > > > >> To: Hannes Reinecke > > > > > > > > > >> Cc: Kashyap Desai; Jens Axboe; > > > > > > > > > >> linux-bl...@vger.kernel.org; Christoph Hellwig; Mike > > > > > > > > > >> Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > > > > > > > > > > Sandoval; > > > > > > > > > >> Martin K . Petersen; James Bottomley; Christoph > > > > > > > > > >> Hellwig; Don Brace; > > > > > > > > > > Peter > > > > > > > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > > > > > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support > > > > > > > > > &g
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Tue, Feb 13, 2018 at 12:05:14AM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Ming Lei [mailto:ming@redhat.com] > > Sent: Sunday, February 11, 2018 11:01 AM > > To: Kashyap Desai > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter > > Rivera; Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > On Sat, Feb 10, 2018 at 09:00:57AM +0800, Ming Lei wrote: > > > Hi Kashyap, > > > > > > On Fri, Feb 09, 2018 at 02:12:16PM +0530, Kashyap Desai wrote: > > > > > -Original Message- > > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > > Sent: Friday, February 9, 2018 11:01 AM > > > > > To: Kashyap Desai > > > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > > > Easi; Omar > > > > Sandoval; > > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > > Brace; > > > > Peter > > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > introduce force_blk_mq > > > > > > > > > > On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > > > > > > -Original Message- > > > > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > > > > Sent: Thursday, February 8, 2018 10:23 PM > > > > > > > To: Hannes Reinecke > > > > > > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; > > > > > > > Arun Easi; Omar > > > > > > Sandoval; > > > > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > > > > Brace; > > > > > > Peter > > > > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > > > introduce force_blk_mq > > > > > > > > > > > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke > wrote: > > > > > > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > > > > > > >> -Original Message- > > > > > > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > > > > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > > > > > > >> To: Hannes Reinecke > > > > > > > > >> Cc: Kashyap Desai; Jens Axboe; > > > > > > > > >> linux-bl...@vger.kernel.org; Christoph Hellwig; Mike > > > > > > > > >> Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > > > > > > > > > Sandoval; > > > > > > > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; > > > > > > > > >> Don Brace; > > > > > > > > > Peter > > > > > > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > > > > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global > > > > > > > > >> tags & introduce force_blk_mq > > > > > > > > >> > > > > > > > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke > > > > wrote: > > > > > > > > >>> Hi all, > > > > > > > > >>> > > > > > > > > >>> [ .. ] > > > > > > > > >>>>> > > > > > > > > >>>>> Could you share us your patch for enabling > > > > > > > > >>>>> global_tags/MQ on > > > > > > > > >>>> megaraid_sas > > > > > > > > >>>>> so that I can reproduce your test? > > > > > > > > >>>>> > > >
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Sunday, February 11, 2018 11:01 AM > To: Kashyap Desai > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > On Sat, Feb 10, 2018 at 09:00:57AM +0800, Ming Lei wrote: > > Hi Kashyap, > > > > On Fri, Feb 09, 2018 at 02:12:16PM +0530, Kashyap Desai wrote: > > > > -Original Message- > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > Sent: Friday, February 9, 2018 11:01 AM > > > > To: Kashyap Desai > > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > > Easi; Omar > > > Sandoval; > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > Brace; > > > Peter > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > introduce force_blk_mq > > > > > > > > On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > > > > > -Original Message- > > > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > > > Sent: Thursday, February 8, 2018 10:23 PM > > > > > > To: Hannes Reinecke > > > > > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; > > > > > > Arun Easi; Omar > > > > > Sandoval; > > > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > > > Brace; > > > > > Peter > > > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > > introduce force_blk_mq > > > > > > > > > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > > > > > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > > > > > >> -Original Message----- > > > > > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > > > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > > > > > >> To: Hannes Reinecke > > > > > > > >> Cc: Kashyap Desai; Jens Axboe; > > > > > > > >> linux-bl...@vger.kernel.org; Christoph Hellwig; Mike > > > > > > > >> Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > > > > > > > > Sandoval; > > > > > > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; > > > > > > > >> Don Brace; > > > > > > > > Peter > > > > > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > > > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global > > > > > > > >> tags & introduce force_blk_mq > > > > > > > >> > > > > > > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke > > > wrote: > > > > > > > >>> Hi all, > > > > > > > >>> > > > > > > > >>> [ .. ] > > > > > > > >>>>> > > > > > > > >>>>> Could you share us your patch for enabling > > > > > > > >>>>> global_tags/MQ on > > > > > > > >>>> megaraid_sas > > > > > > > >>>>> so that I can reproduce your test? > > > > > > > >>>>> > > > > > > > >>>>>> See below perf top data. "bt_iter" is consuming 4 > > > > > > > >>>>>> times more > > > > > CPU. > > > > > > > >>>>> > > > > > > > >>>>> Could you share us what the IOPS/CPU utilization > > > > > > > >>>>> effect is after > > > >
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On Sat, Feb 10, 2018 at 09:00:57AM +0800, Ming Lei wrote: > Hi Kashyap, > > On Fri, Feb 09, 2018 at 02:12:16PM +0530, Kashyap Desai wrote: > > > -Original Message- > > > From: Ming Lei [mailto:ming@redhat.com] > > > Sent: Friday, February 9, 2018 11:01 AM > > > To: Kashyap Desai > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > > Sandoval; > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > Peter > > > Rivera; Paolo Bonzini; Laurence Oberman > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > > force_blk_mq > > > > > > On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > > > > -Original Message- > > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > > Sent: Thursday, February 8, 2018 10:23 PM > > > > > To: Hannes Reinecke > > > > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > > > Easi; Omar > > > > Sandoval; > > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > > > Peter > > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > introduce force_blk_mq > > > > > > > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > > > > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > > > > >> -Original Message- > > > > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > > > > >> To: Hannes Reinecke > > > > > > >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > > >> Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; > > > > > > >> Arun Easi; Omar > > > > > > > Sandoval; > > > > > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > > > >> Brace; > > > > > > > Peter > > > > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > > >> introduce force_blk_mq > > > > > > >> > > > > > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke > > wrote: > > > > > > >>> Hi all, > > > > > > >>> > > > > > > >>> [ .. ] > > > > > > >>>>> > > > > > > >>>>> Could you share us your patch for enabling global_tags/MQ on > > > > > > >>>> megaraid_sas > > > > > > >>>>> so that I can reproduce your test? > > > > > > >>>>> > > > > > > >>>>>> See below perf top data. "bt_iter" is consuming 4 times > > > > > > >>>>>> more > > > > CPU. > > > > > > >>>>> > > > > > > >>>>> Could you share us what the IOPS/CPU utilization effect is > > > > > > >>>>> after > > > > > > >>>> applying the > > > > > > >>>>> patch V2? And your test script? > > > > > > >>>> Regarding CPU utilization, I need to test one more time. > > > > > > >>>> Currently system is in used. > > > > > > >>>> > > > > > > >>>> I run below fio test on total 24 SSDs expander attached. > > > > > > >>>> > > > > > > >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > > > > > >>>> --ioengine=libaio --rw=randread > > > > > > >>>> > > > > > > >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. > > > > > > >>>> > > > > > > >>> This is basically what we've seen with earlier iterations. > >
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Fri, Feb 09, 2018 at 02:12:16PM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Ming Lei [mailto:ming@redhat.com] > > Sent: Friday, February 9, 2018 11:01 AM > > To: Kashyap Desai > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter > > Rivera; Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > > > -Original Message- > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > Sent: Thursday, February 8, 2018 10:23 PM > > > > To: Hannes Reinecke > > > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > > Easi; Omar > > > Sandoval; > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > > Peter > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > introduce force_blk_mq > > > > > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > > > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > > > >> -Original Message- > > > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > > > >> To: Hannes Reinecke > > > > > >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > > >> Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; > > > > > >> Arun Easi; Omar > > > > > > Sandoval; > > > > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > > >> Brace; > > > > > > Peter > > > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > > >> introduce force_blk_mq > > > > > >> > > > > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke > wrote: > > > > > >>> Hi all, > > > > > >>> > > > > > >>> [ .. ] > > > > > >>>>> > > > > > >>>>> Could you share us your patch for enabling global_tags/MQ on > > > > > >>>> megaraid_sas > > > > > >>>>> so that I can reproduce your test? > > > > > >>>>> > > > > > >>>>>> See below perf top data. "bt_iter" is consuming 4 times > > > > > >>>>>> more > > > CPU. > > > > > >>>>> > > > > > >>>>> Could you share us what the IOPS/CPU utilization effect is > > > > > >>>>> after > > > > > >>>> applying the > > > > > >>>>> patch V2? And your test script? > > > > > >>>> Regarding CPU utilization, I need to test one more time. > > > > > >>>> Currently system is in used. > > > > > >>>> > > > > > >>>> I run below fio test on total 24 SSDs expander attached. > > > > > >>>> > > > > > >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > > > > >>>> --ioengine=libaio --rw=randread > > > > > >>>> > > > > > >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. > > > > > >>>> > > > > > >>> This is basically what we've seen with earlier iterations. > > > > > >> > > > > > >> Hi Hannes, > > > > > >> > > > > > >> As I mentioned in another mail[1], Kashyap's patch has a big > > > > > >> issue, > > > > > > which > > > > > >> causes only reply queue 0 used. > > > > > >> > > > > > >> [1] https://marc.info/?l=linux-scsi
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Friday, February 9, 2018 11:01 AM > To: Kashyap Desai > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > > -Original Message- > > > From: Ming Lei [mailto:ming@redhat.com] > > > Sent: Thursday, February 8, 2018 10:23 PM > > > To: Hannes Reinecke > > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > Easi; Omar > > Sandoval; > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > Peter > > > Rivera; Paolo Bonzini; Laurence Oberman > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > introduce force_blk_mq > > > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > > >> -Original Message- > > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > > >> To: Hannes Reinecke > > > > >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > > >> Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; > > > > >> Arun Easi; Omar > > > > > Sandoval; > > > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don > > > > >> Brace; > > > > > Peter > > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > >> introduce force_blk_mq > > > > >> > > > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > > > > >>> Hi all, > > > > >>> > > > > >>> [ .. ] > > > > >>>>> > > > > >>>>> Could you share us your patch for enabling global_tags/MQ on > > > > >>>> megaraid_sas > > > > >>>>> so that I can reproduce your test? > > > > >>>>> > > > > >>>>>> See below perf top data. "bt_iter" is consuming 4 times > > > > >>>>>> more > > CPU. > > > > >>>>> > > > > >>>>> Could you share us what the IOPS/CPU utilization effect is > > > > >>>>> after > > > > >>>> applying the > > > > >>>>> patch V2? And your test script? > > > > >>>> Regarding CPU utilization, I need to test one more time. > > > > >>>> Currently system is in used. > > > > >>>> > > > > >>>> I run below fio test on total 24 SSDs expander attached. > > > > >>>> > > > > >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > > > >>>> --ioengine=libaio --rw=randread > > > > >>>> > > > > >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. > > > > >>>> > > > > >>> This is basically what we've seen with earlier iterations. > > > > >> > > > > >> Hi Hannes, > > > > >> > > > > >> As I mentioned in another mail[1], Kashyap's patch has a big > > > > >> issue, > > > > > which > > > > >> causes only reply queue 0 used. > > > > >> > > > > >> [1] https://marc.info/?l=linux-scsi=151793204014631=2 > > > > >> > > > > >> So could you guys run your performance test again after fixing > > > > >> the > > > > > patch? > > > > > > > > > > Ming - > > > > > > > > > > I tried after change you requested. Performance drop is still > > unresolved. > > > > > From 1.6 M IOPS to 770K IOPS. > > > > > > > &g
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On Fri, Feb 09, 2018 at 10:28:23AM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Ming Lei [mailto:ming@redhat.com] > > Sent: Thursday, February 8, 2018 10:23 PM > > To: Hannes Reinecke > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter > > Rivera; Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > > >> -Original Message- > > > >> From: Ming Lei [mailto:ming@redhat.com] > > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > > >> To: Hannes Reinecke > > > >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > > >> Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > >> Easi; Omar > > > > Sandoval; > > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > > > Peter > > > >> Rivera; Paolo Bonzini; Laurence Oberman > > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > >> introduce force_blk_mq > > > >> > > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > > > >>> Hi all, > > > >>> > > > >>> [ .. ] > > > >>>>> > > > >>>>> Could you share us your patch for enabling global_tags/MQ on > > > >>>> megaraid_sas > > > >>>>> so that I can reproduce your test? > > > >>>>> > > > >>>>>> See below perf top data. "bt_iter" is consuming 4 times more > CPU. > > > >>>>> > > > >>>>> Could you share us what the IOPS/CPU utilization effect is after > > > >>>> applying the > > > >>>>> patch V2? And your test script? > > > >>>> Regarding CPU utilization, I need to test one more time. > > > >>>> Currently system is in used. > > > >>>> > > > >>>> I run below fio test on total 24 SSDs expander attached. > > > >>>> > > > >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > > >>>> --ioengine=libaio --rw=randread > > > >>>> > > > >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. > > > >>>> > > > >>> This is basically what we've seen with earlier iterations. > > > >> > > > >> Hi Hannes, > > > >> > > > >> As I mentioned in another mail[1], Kashyap's patch has a big issue, > > > > which > > > >> causes only reply queue 0 used. > > > >> > > > >> [1] https://marc.info/?l=linux-scsi=151793204014631=2 > > > >> > > > >> So could you guys run your performance test again after fixing the > > > > patch? > > > > > > > > Ming - > > > > > > > > I tried after change you requested. Performance drop is still > unresolved. > > > > From 1.6 M IOPS to 770K IOPS. > > > > > > > > See below data. All 24 reply queue is in used correctly. > > > > > > > > IRQs / 1 second(s) > > > > IRQ# TOTAL NODE0 NODE1 NAME > > > > 360 16422 0 16422 IR-PCI-MSI 70254653-edge megasas > > > > 364 15980 0 15980 IR-PCI-MSI 70254657-edge megasas > > > > 362 15979 0 15979 IR-PCI-MSI 70254655-edge megasas > > > > 345 15696 0 15696 IR-PCI-MSI 70254638-edge megasas > > > > 341 15659 0 15659 IR-PCI-MSI 70254634-edge megasas > > > > 369 15656 0 15656 IR-PCI-MSI 70254662-edge megasas > > > > 359 15650 0 15650 IR-PCI-MSI 70254652-edge megasas > > > > 358 15596 0 15596 IR-PCI-MSI 70254651-edge megasas > > > > 350 15574 0 15574 IR-PCI-MSI 70254643-edge megasas > > > > 342 15532 0 15532 IR-PCI-MSI 70254635-edge megasas > > > > 344 15527 0 15527 IR-PCI-MSI 70254637-edge megasas > > > > 346 15
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Thursday, February 8, 2018 10:23 PM > To: Hannes Reinecke > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > > >> -Original Message- > > >> From: Ming Lei [mailto:ming@redhat.com] > > >> Sent: Wednesday, February 7, 2018 5:53 PM > > >> To: Hannes Reinecke > > >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; > > >> Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > >> Easi; Omar > > > Sandoval; > > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > > Peter > > >> Rivera; Paolo Bonzini; Laurence Oberman > > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > >> introduce force_blk_mq > > >> > > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > > >>> Hi all, > > >>> > > >>> [ .. ] > > >>>>> > > >>>>> Could you share us your patch for enabling global_tags/MQ on > > >>>> megaraid_sas > > >>>>> so that I can reproduce your test? > > >>>>> > > >>>>>> See below perf top data. "bt_iter" is consuming 4 times more CPU. > > >>>>> > > >>>>> Could you share us what the IOPS/CPU utilization effect is after > > >>>> applying the > > >>>>> patch V2? And your test script? > > >>>> Regarding CPU utilization, I need to test one more time. > > >>>> Currently system is in used. > > >>>> > > >>>> I run below fio test on total 24 SSDs expander attached. > > >>>> > > >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > >>>> --ioengine=libaio --rw=randread > > >>>> > > >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. > > >>>> > > >>> This is basically what we've seen with earlier iterations. > > >> > > >> Hi Hannes, > > >> > > >> As I mentioned in another mail[1], Kashyap's patch has a big issue, > > > which > > >> causes only reply queue 0 used. > > >> > > >> [1] https://marc.info/?l=linux-scsi=151793204014631=2 > > >> > > >> So could you guys run your performance test again after fixing the > > > patch? > > > > > > Ming - > > > > > > I tried after change you requested. Performance drop is still unresolved. > > > From 1.6 M IOPS to 770K IOPS. > > > > > > See below data. All 24 reply queue is in used correctly. > > > > > > IRQs / 1 second(s) > > > IRQ# TOTAL NODE0 NODE1 NAME > > > 360 16422 0 16422 IR-PCI-MSI 70254653-edge megasas > > > 364 15980 0 15980 IR-PCI-MSI 70254657-edge megasas > > > 362 15979 0 15979 IR-PCI-MSI 70254655-edge megasas > > > 345 15696 0 15696 IR-PCI-MSI 70254638-edge megasas > > > 341 15659 0 15659 IR-PCI-MSI 70254634-edge megasas > > > 369 15656 0 15656 IR-PCI-MSI 70254662-edge megasas > > > 359 15650 0 15650 IR-PCI-MSI 70254652-edge megasas > > > 358 15596 0 15596 IR-PCI-MSI 70254651-edge megasas > > > 350 15574 0 15574 IR-PCI-MSI 70254643-edge megasas > > > 342 15532 0 15532 IR-PCI-MSI 70254635-edge megasas > > > 344 15527 0 15527 IR-PCI-MSI 70254637-edge megasas > > > 346 15485 0 15485 IR-PCI-MSI 70254639-edge megasas > > > 361 15482 0 15482 IR-PCI-MSI 70254654-edge megasas > > > 348 15467 0 15467 IR-PCI-MSI 70254641-edge megasas > > > 368 15463 0 15463 IR-PCI-MSI 70254661-edge megasas > > > 354 15420 0 15420 IR-PCI-MSI 70254647-edge megasas > > > 351 15378 0 15378 IR-PCI-MSI 70254644-edge megasas > > > 352 15377 0 15377 IR-PCI-MSI 70254645-edge
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On Thu, Feb 08, 2018 at 08:00:29AM +0100, Hannes Reinecke wrote: > On 02/07/2018 03:14 PM, Kashyap Desai wrote: > >> -Original Message- > >> From: Ming Lei [mailto:ming@redhat.com] > >> Sent: Wednesday, February 7, 2018 5:53 PM > >> To: Hannes Reinecke > >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > >> Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > > Sandoval; > >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > Peter > >> Rivera; Paolo Bonzini; Laurence Oberman > >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > >> force_blk_mq > >> > >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > >>> Hi all, > >>> > >>> [ .. ] > >>>>> > >>>>> Could you share us your patch for enabling global_tags/MQ on > >>>> megaraid_sas > >>>>> so that I can reproduce your test? > >>>>> > >>>>>> See below perf top data. "bt_iter" is consuming 4 times more CPU. > >>>>> > >>>>> Could you share us what the IOPS/CPU utilization effect is after > >>>> applying the > >>>>> patch V2? And your test script? > >>>> Regarding CPU utilization, I need to test one more time. Currently > >>>> system is in used. > >>>> > >>>> I run below fio test on total 24 SSDs expander attached. > >>>> > >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > >>>> --ioengine=libaio --rw=randread > >>>> > >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. > >>>> > >>> This is basically what we've seen with earlier iterations. > >> > >> Hi Hannes, > >> > >> As I mentioned in another mail[1], Kashyap's patch has a big issue, > > which > >> causes only reply queue 0 used. > >> > >> [1] https://marc.info/?l=linux-scsi=151793204014631=2 > >> > >> So could you guys run your performance test again after fixing the > > patch? > > > > Ming - > > > > I tried after change you requested. Performance drop is still unresolved. > > From 1.6 M IOPS to 770K IOPS. > > > > See below data. All 24 reply queue is in used correctly. > > > > IRQs / 1 second(s) > > IRQ# TOTAL NODE0 NODE1 NAME > > 360 16422 0 16422 IR-PCI-MSI 70254653-edge megasas > > 364 15980 0 15980 IR-PCI-MSI 70254657-edge megasas > > 362 15979 0 15979 IR-PCI-MSI 70254655-edge megasas > > 345 15696 0 15696 IR-PCI-MSI 70254638-edge megasas > > 341 15659 0 15659 IR-PCI-MSI 70254634-edge megasas > > 369 15656 0 15656 IR-PCI-MSI 70254662-edge megasas > > 359 15650 0 15650 IR-PCI-MSI 70254652-edge megasas > > 358 15596 0 15596 IR-PCI-MSI 70254651-edge megasas > > 350 15574 0 15574 IR-PCI-MSI 70254643-edge megasas > > 342 15532 0 15532 IR-PCI-MSI 70254635-edge megasas > > 344 15527 0 15527 IR-PCI-MSI 70254637-edge megasas > > 346 15485 0 15485 IR-PCI-MSI 70254639-edge megasas > > 361 15482 0 15482 IR-PCI-MSI 70254654-edge megasas > > 348 15467 0 15467 IR-PCI-MSI 70254641-edge megasas > > 368 15463 0 15463 IR-PCI-MSI 70254661-edge megasas > > 354 15420 0 15420 IR-PCI-MSI 70254647-edge megasas > > 351 15378 0 15378 IR-PCI-MSI 70254644-edge megasas > > 352 15377 0 15377 IR-PCI-MSI 70254645-edge megasas > > 356 15348 0 15348 IR-PCI-MSI 70254649-edge megasas > > 337 15344 0 15344 IR-PCI-MSI 70254630-edge megasas > > 343 15320 0 15320 IR-PCI-MSI 70254636-edge megasas > > 355 15266 0 15266 IR-PCI-MSI 70254648-edge megasas > > 335 15247 0 15247 IR-PCI-MSI 70254628-edge megasas > > 363 15233 0 15233 IR-PCI-MSI 70254656-edge megasas > > > > > > Average:CPU %usr %nice %sys %iowait%steal > > %irq %soft%guest%gnice %idle > > Average: 18 3.80 0.00 14.78 10.08 0.00 > > 0.00 4.01 0.00 0.00 67.33 > > Average: 19 3.26 0.00 15.35 10.62 0.00 > > 0.00 4.03 0.00 0.00 66.74 > > Average: 20 3.42 0.00 14.57 10.67 0.00 > > 0.00
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On 02/07/2018 03:14 PM, Kashyap Desai wrote: >> -Original Message- >> From: Ming Lei [mailto:ming@redhat.com] >> Sent: Wednesday, February 7, 2018 5:53 PM >> To: Hannes Reinecke >> Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; Christoph >> Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; >> Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter >> Rivera; Paolo Bonzini; Laurence Oberman >> Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce >> force_blk_mq >> >> On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: >>> Hi all, >>> >>> [ .. ] >>>>> >>>>> Could you share us your patch for enabling global_tags/MQ on >>>> megaraid_sas >>>>> so that I can reproduce your test? >>>>> >>>>>> See below perf top data. "bt_iter" is consuming 4 times more CPU. >>>>> >>>>> Could you share us what the IOPS/CPU utilization effect is after >>>> applying the >>>>> patch V2? And your test script? >>>> Regarding CPU utilization, I need to test one more time. Currently >>>> system is in used. >>>> >>>> I run below fio test on total 24 SSDs expander attached. >>>> >>>> numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k >>>> --ioengine=libaio --rw=randread >>>> >>>> Performance dropped from 1.6 M IOPs to 770K IOPs. >>>> >>> This is basically what we've seen with earlier iterations. >> >> Hi Hannes, >> >> As I mentioned in another mail[1], Kashyap's patch has a big issue, > which >> causes only reply queue 0 used. >> >> [1] https://marc.info/?l=linux-scsi=151793204014631=2 >> >> So could you guys run your performance test again after fixing the > patch? > > Ming - > > I tried after change you requested. Performance drop is still unresolved. > From 1.6 M IOPS to 770K IOPS. > > See below data. All 24 reply queue is in used correctly. > > IRQs / 1 second(s) > IRQ# TOTAL NODE0 NODE1 NAME > 360 16422 0 16422 IR-PCI-MSI 70254653-edge megasas > 364 15980 0 15980 IR-PCI-MSI 70254657-edge megasas > 362 15979 0 15979 IR-PCI-MSI 70254655-edge megasas > 345 15696 0 15696 IR-PCI-MSI 70254638-edge megasas > 341 15659 0 15659 IR-PCI-MSI 70254634-edge megasas > 369 15656 0 15656 IR-PCI-MSI 70254662-edge megasas > 359 15650 0 15650 IR-PCI-MSI 70254652-edge megasas > 358 15596 0 15596 IR-PCI-MSI 70254651-edge megasas > 350 15574 0 15574 IR-PCI-MSI 70254643-edge megasas > 342 15532 0 15532 IR-PCI-MSI 70254635-edge megasas > 344 15527 0 15527 IR-PCI-MSI 70254637-edge megasas > 346 15485 0 15485 IR-PCI-MSI 70254639-edge megasas > 361 15482 0 15482 IR-PCI-MSI 70254654-edge megasas > 348 15467 0 15467 IR-PCI-MSI 70254641-edge megasas > 368 15463 0 15463 IR-PCI-MSI 70254661-edge megasas > 354 15420 0 15420 IR-PCI-MSI 70254647-edge megasas > 351 15378 0 15378 IR-PCI-MSI 70254644-edge megasas > 352 15377 0 15377 IR-PCI-MSI 70254645-edge megasas > 356 15348 0 15348 IR-PCI-MSI 70254649-edge megasas > 337 15344 0 15344 IR-PCI-MSI 70254630-edge megasas > 343 15320 0 15320 IR-PCI-MSI 70254636-edge megasas > 355 15266 0 15266 IR-PCI-MSI 70254648-edge megasas > 335 15247 0 15247 IR-PCI-MSI 70254628-edge megasas > 363 15233 0 15233 IR-PCI-MSI 70254656-edge megasas > > > Average:CPU %usr %nice %sys %iowait%steal > %irq %soft%guest%gnice %idle > Average: 18 3.80 0.00 14.78 10.08 0.00 > 0.00 4.01 0.00 0.00 67.33 > Average: 19 3.26 0.00 15.35 10.62 0.00 > 0.00 4.03 0.00 0.00 66.74 > Average: 20 3.42 0.00 14.57 10.67 0.00 > 0.00 3.84 0.00 0.00 67.50 > Average: 21 3.19 0.00 15.60 10.75 0.00 > 0.00 4.16 0.00 0.00 66.30 > Average: 22 3.58 0.00 15.15 10.66 0.00 > 0.00 3.51 0.00 0.00 67.11 > Average: 23 3.34 0.00 15.36 10.63 0.00 > 0.00 4.17 0.00 0.00 66.50 > Average: 24 3.50 0.00 14.58 10.93 0.00 > 0.00 3.85 0.00 0.00
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Wed, Feb 07, 2018 at 07:44:04PM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Ming Lei [mailto:ming@redhat.com] > > Sent: Wednesday, February 7, 2018 5:53 PM > > To: Hannes Reinecke > > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter > > Rivera; Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > > > Hi all, > > > > > > [ .. ] > > > >> > > > >> Could you share us your patch for enabling global_tags/MQ on > > > > megaraid_sas > > > >> so that I can reproduce your test? > > > >> > > > >>> See below perf top data. "bt_iter" is consuming 4 times more CPU. > > > >> > > > >> Could you share us what the IOPS/CPU utilization effect is after > > > > applying the > > > >> patch V2? And your test script? > > > > Regarding CPU utilization, I need to test one more time. Currently > > > > system is in used. > > > > > > > > I run below fio test on total 24 SSDs expander attached. > > > > > > > > numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > > > --ioengine=libaio --rw=randread > > > > > > > > Performance dropped from 1.6 M IOPs to 770K IOPs. > > > > > > > This is basically what we've seen with earlier iterations. > > > > Hi Hannes, > > > > As I mentioned in another mail[1], Kashyap's patch has a big issue, > which > > causes only reply queue 0 used. > > > > [1] https://marc.info/?l=linux-scsi=151793204014631=2 > > > > So could you guys run your performance test again after fixing the > patch? > > Ming - > > I tried after change you requested. Performance drop is still unresolved. > From 1.6 M IOPS to 770K IOPS. > > See below data. All 24 reply queue is in used correctly. > > IRQs / 1 second(s) > IRQ# TOTAL NODE0 NODE1 NAME > 360 16422 0 16422 IR-PCI-MSI 70254653-edge megasas > 364 15980 0 15980 IR-PCI-MSI 70254657-edge megasas > 362 15979 0 15979 IR-PCI-MSI 70254655-edge megasas > 345 15696 0 15696 IR-PCI-MSI 70254638-edge megasas > 341 15659 0 15659 IR-PCI-MSI 70254634-edge megasas > 369 15656 0 15656 IR-PCI-MSI 70254662-edge megasas > 359 15650 0 15650 IR-PCI-MSI 70254652-edge megasas > 358 15596 0 15596 IR-PCI-MSI 70254651-edge megasas > 350 15574 0 15574 IR-PCI-MSI 70254643-edge megasas > 342 15532 0 15532 IR-PCI-MSI 70254635-edge megasas > 344 15527 0 15527 IR-PCI-MSI 70254637-edge megasas > 346 15485 0 15485 IR-PCI-MSI 70254639-edge megasas > 361 15482 0 15482 IR-PCI-MSI 70254654-edge megasas > 348 15467 0 15467 IR-PCI-MSI 70254641-edge megasas > 368 15463 0 15463 IR-PCI-MSI 70254661-edge megasas > 354 15420 0 15420 IR-PCI-MSI 70254647-edge megasas > 351 15378 0 15378 IR-PCI-MSI 70254644-edge megasas > 352 15377 0 15377 IR-PCI-MSI 70254645-edge megasas > 356 15348 0 15348 IR-PCI-MSI 70254649-edge megasas > 337 15344 0 15344 IR-PCI-MSI 70254630-edge megasas > 343 15320 0 15320 IR-PCI-MSI 70254636-edge megasas > 355 15266 0 15266 IR-PCI-MSI 70254648-edge megasas > 335 15247 0 15247 IR-PCI-MSI 70254628-edge megasas > 363 15233 0 15233 IR-PCI-MSI 70254656-edge megasas > > > Average:CPU %usr %nice %sys %iowait%steal > %irq %soft%guest%gnice %idle > Average: 18 3.80 0.00 14.78 10.08 0.00 > 0.00 4.01 0.00 0.00 67.33 > Average: 19 3.26 0.00 15.35 10.62 0.00 > 0.00 4.03 0.00 0.00 66.74 > Average: 20 3.42 0.00 14.57 10.67 0.00 > 0.00 3.84 0.00 0.00 67.50 > Average: 21 3.19 0.00 15.60 10.75 0.00 > 0.00 4.16 0.00 0.00 66.30 > Average: 22 3.58 0.00 15.15 10.66 0.00 > 0.00 3.51 0.00 0.00 67.11 > Average: 23 3.34 0.00 15.36 10.63 0.00 > 0.00 4.17 0.00 0.00 66.50
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Wednesday, February 7, 2018 5:53 PM > To: Hannes Reinecke > Cc: Kashyap Desai; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > > Hi all, > > > > [ .. ] > > >> > > >> Could you share us your patch for enabling global_tags/MQ on > > > megaraid_sas > > >> so that I can reproduce your test? > > >> > > >>> See below perf top data. "bt_iter" is consuming 4 times more CPU. > > >> > > >> Could you share us what the IOPS/CPU utilization effect is after > > > applying the > > >> patch V2? And your test script? > > > Regarding CPU utilization, I need to test one more time. Currently > > > system is in used. > > > > > > I run below fio test on total 24 SSDs expander attached. > > > > > > numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > > --ioengine=libaio --rw=randread > > > > > > Performance dropped from 1.6 M IOPs to 770K IOPs. > > > > > This is basically what we've seen with earlier iterations. > > Hi Hannes, > > As I mentioned in another mail[1], Kashyap's patch has a big issue, which > causes only reply queue 0 used. > > [1] https://marc.info/?l=linux-scsi=151793204014631=2 > > So could you guys run your performance test again after fixing the patch? Ming - I tried after change you requested. Performance drop is still unresolved. >From 1.6 M IOPS to 770K IOPS. See below data. All 24 reply queue is in used correctly. IRQs / 1 second(s) IRQ# TOTAL NODE0 NODE1 NAME 360 16422 0 16422 IR-PCI-MSI 70254653-edge megasas 364 15980 0 15980 IR-PCI-MSI 70254657-edge megasas 362 15979 0 15979 IR-PCI-MSI 70254655-edge megasas 345 15696 0 15696 IR-PCI-MSI 70254638-edge megasas 341 15659 0 15659 IR-PCI-MSI 70254634-edge megasas 369 15656 0 15656 IR-PCI-MSI 70254662-edge megasas 359 15650 0 15650 IR-PCI-MSI 70254652-edge megasas 358 15596 0 15596 IR-PCI-MSI 70254651-edge megasas 350 15574 0 15574 IR-PCI-MSI 70254643-edge megasas 342 15532 0 15532 IR-PCI-MSI 70254635-edge megasas 344 15527 0 15527 IR-PCI-MSI 70254637-edge megasas 346 15485 0 15485 IR-PCI-MSI 70254639-edge megasas 361 15482 0 15482 IR-PCI-MSI 70254654-edge megasas 348 15467 0 15467 IR-PCI-MSI 70254641-edge megasas 368 15463 0 15463 IR-PCI-MSI 70254661-edge megasas 354 15420 0 15420 IR-PCI-MSI 70254647-edge megasas 351 15378 0 15378 IR-PCI-MSI 70254644-edge megasas 352 15377 0 15377 IR-PCI-MSI 70254645-edge megasas 356 15348 0 15348 IR-PCI-MSI 70254649-edge megasas 337 15344 0 15344 IR-PCI-MSI 70254630-edge megasas 343 15320 0 15320 IR-PCI-MSI 70254636-edge megasas 355 15266 0 15266 IR-PCI-MSI 70254648-edge megasas 335 15247 0 15247 IR-PCI-MSI 70254628-edge megasas 363 15233 0 15233 IR-PCI-MSI 70254656-edge megasas Average:CPU %usr %nice %sys %iowait%steal %irq %soft%guest%gnice %idle Average: 18 3.80 0.00 14.78 10.08 0.00 0.00 4.01 0.00 0.00 67.33 Average: 19 3.26 0.00 15.35 10.62 0.00 0.00 4.03 0.00 0.00 66.74 Average: 20 3.42 0.00 14.57 10.67 0.00 0.00 3.84 0.00 0.00 67.50 Average: 21 3.19 0.00 15.60 10.75 0.00 0.00 4.16 0.00 0.00 66.30 Average: 22 3.58 0.00 15.15 10.66 0.00 0.00 3.51 0.00 0.00 67.11 Average: 23 3.34 0.00 15.36 10.63 0.00 0.00 4.17 0.00 0.00 66.50 Average: 24 3.50 0.00 14.58 10.93 0.00 0.00 3.85 0.00 0.00 67.13 Average: 25 3.20 0.00 14.68 10.86 0.00 0.00 4.31 0.00 0.00 66.95 Average: 26 3.27 0.00 14.80 10.70 0.00 0.00 3.68 0.00 0.00 67.55 Average: 27 3.58 0.00 15.36 10.80 0.00 0.00 3.79 0.00 0.00 66.48 Average: 28 3.46 0.00 15.17 10.46 0.00 0.00 3.32 0.00 0.00
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On Wed, Feb 07, 2018 at 07:50:21AM +0100, Hannes Reinecke wrote: > Hi all, > > [ .. ] > >> > >> Could you share us your patch for enabling global_tags/MQ on > > megaraid_sas > >> so that I can reproduce your test? > >> > >>> See below perf top data. "bt_iter" is consuming 4 times more CPU. > >> > >> Could you share us what the IOPS/CPU utilization effect is after > > applying the > >> patch V2? And your test script? > > Regarding CPU utilization, I need to test one more time. Currently system > > is in used. > > > > I run below fio test on total 24 SSDs expander attached. > > > > numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > > --ioengine=libaio --rw=randread > > > > Performance dropped from 1.6 M IOPs to 770K IOPs. > > > This is basically what we've seen with earlier iterations. Hi Hannes, As I mentioned in another mail[1], Kashyap's patch has a big issue, which causes only reply queue 0 used. [1] https://marc.info/?l=linux-scsi=151793204014631=2 So could you guys run your performance test again after fixing the patch? Thanks, Ming
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi all, [ .. ] >> >> Could you share us your patch for enabling global_tags/MQ on > megaraid_sas >> so that I can reproduce your test? >> >>> See below perf top data. "bt_iter" is consuming 4 times more CPU. >> >> Could you share us what the IOPS/CPU utilization effect is after > applying the >> patch V2? And your test script? > Regarding CPU utilization, I need to test one more time. Currently system > is in used. > > I run below fio test on total 24 SSDs expander attached. > > numactl -N 1 fio jbod.fio --rw=randread --iodepth=64 --bs=4k > --ioengine=libaio --rw=randread > > Performance dropped from 1.6 M IOPs to 770K IOPs. > This is basically what we've seen with earlier iterations. >> >> In theory, it shouldn't, because the HBA only supports HBA wide tags, >> that means the allocation has to share a HBA wide sbitmap no matte>> if >> global tags is used or not. >> >> Anyway, I will take a look at the performance test and data. >> >> >> Thanks, >> Ming > > > Megaraid_sas version of shared tag set. > Whee; thanks for that. I've just finished a patchset moving megarai_sas_fusion to embedded commands (and cutting down the size of 'struct megasas_cmd_fusion' by half :-), so that will come in just handy. Will give it a spin. Cheers, Hannes -- Dr. Hannes ReineckeTeamlead Storage & Networking h...@suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Tue, Feb 06, 2018 at 07:57:35PM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Ming Lei [mailto:ming@redhat.com] > > Sent: Tuesday, February 6, 2018 6:02 PM > > To: Kashyap Desai > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter > > Rivera; Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > Hi Kashyap, > > > > On Tue, Feb 06, 2018 at 04:59:51PM +0530, Kashyap Desai wrote: > > > > -Original Message- > > > > From: Ming Lei [mailto:ming@redhat.com] > > > > Sent: Tuesday, February 6, 2018 1:35 PM > > > > To: Kashyap Desai > > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; > > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > > Easi; Omar > > > Sandoval; > > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > > Peter > > > > Rivera; Paolo Bonzini; Laurence Oberman > > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > > introduce force_blk_mq > > > > > > > > Hi Kashyap, > > > > > > > > On Tue, Feb 06, 2018 at 11:33:50AM +0530, Kashyap Desai wrote: > > > > > > > We still have more than one reply queue ending up completion > > > > > > > one > > > CPU. > > > > > > > > > > > > pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that > > > > > > means smp_affinity_enable has to be set as 1, but seems it is > > > > > > the default > > > > > setting. > > > > > > > > > > > > Please see kernel/irq/affinity.c, especially > > > > > > irq_calc_affinity_vectors() > > > > > which > > > > > > figures out an optimal number of vectors, and the computation is > > > > > > based > > > > > on > > > > > > cpumask_weight(cpu_possible_mask) now. If all offline CPUs are > > > > > > mapped to some of reply queues, these queues won't be active(no > > > > > > request submitted > > > > > to > > > > > > these queues). The mechanism of PCI_IRQ_AFFINITY basically makes > > > > > > sure > > > > > that > > > > > > more than one irq vector won't be handled by one same CPU, and > > > > > > the irq vector spread is done in irq_create_affinity_masks(). > > > > > > > > > > > > > Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver > > > > > > > via module parameter to simulate the issue. We need more > > > > > > > number of Online CPU than reply-queue. > > > > > > > > > > > > IMO, you don't need to simulate the issue, > > > > > > pci_alloc_irq_vectors( > > > > > > PCI_IRQ_AFFINITY) will handle that for you. You can dump the > > > > > > returned > > > > > irq > > > > > > vector number, num_possible_cpus()/num_online_cpus() and each > > > > > > irq vector's affinity assignment. > > > > > > > > > > > > > We may see completion redirected to original CPU because of > > > > > > > "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep > > > > > > > one CPU busy in local ISR routine. > > > > > > > > > > > > Could you dump each irq vector's affinity assignment of your > > > > > > megaraid in > > > > > your > > > > > > test? > > > > > > > > > > To quickly reproduce, I restricted to single MSI-x vector on > > > > > megaraid_sas driver. System has total 16 online CPUs. > > > > > > > > I suggest you don't do the restriction of single MSI-x vector, and > > > > just > > > use the > > > > device supported number of msi-x vector. > > > > > > Hi Ming, CPU lock up is seen even though it is not single msi-x > vector. > > > Actual scenario need some specific topology and server for overnight > test. > >
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Tuesday, February 6, 2018 6:02 PM > To: Kashyap Desai > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > Hi Kashyap, > > On Tue, Feb 06, 2018 at 04:59:51PM +0530, Kashyap Desai wrote: > > > -Original Message- > > > From: Ming Lei [mailto:ming@redhat.com] > > > Sent: Tuesday, February 6, 2018 1:35 PM > > > To: Kashyap Desai > > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; > > > Christoph Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun > > > Easi; Omar > > Sandoval; > > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > > Peter > > > Rivera; Paolo Bonzini; Laurence Oberman > > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & > > > introduce force_blk_mq > > > > > > Hi Kashyap, > > > > > > On Tue, Feb 06, 2018 at 11:33:50AM +0530, Kashyap Desai wrote: > > > > > > We still have more than one reply queue ending up completion > > > > > > one > > CPU. > > > > > > > > > > pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that > > > > > means smp_affinity_enable has to be set as 1, but seems it is > > > > > the default > > > > setting. > > > > > > > > > > Please see kernel/irq/affinity.c, especially > > > > > irq_calc_affinity_vectors() > > > > which > > > > > figures out an optimal number of vectors, and the computation is > > > > > based > > > > on > > > > > cpumask_weight(cpu_possible_mask) now. If all offline CPUs are > > > > > mapped to some of reply queues, these queues won't be active(no > > > > > request submitted > > > > to > > > > > these queues). The mechanism of PCI_IRQ_AFFINITY basically makes > > > > > sure > > > > that > > > > > more than one irq vector won't be handled by one same CPU, and > > > > > the irq vector spread is done in irq_create_affinity_masks(). > > > > > > > > > > > Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver > > > > > > via module parameter to simulate the issue. We need more > > > > > > number of Online CPU than reply-queue. > > > > > > > > > > IMO, you don't need to simulate the issue, > > > > > pci_alloc_irq_vectors( > > > > > PCI_IRQ_AFFINITY) will handle that for you. You can dump the > > > > > returned > > > > irq > > > > > vector number, num_possible_cpus()/num_online_cpus() and each > > > > > irq vector's affinity assignment. > > > > > > > > > > > We may see completion redirected to original CPU because of > > > > > > "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep > > > > > > one CPU busy in local ISR routine. > > > > > > > > > > Could you dump each irq vector's affinity assignment of your > > > > > megaraid in > > > > your > > > > > test? > > > > > > > > To quickly reproduce, I restricted to single MSI-x vector on > > > > megaraid_sas driver. System has total 16 online CPUs. > > > > > > I suggest you don't do the restriction of single MSI-x vector, and > > > just > > use the > > > device supported number of msi-x vector. > > > > Hi Ming, CPU lock up is seen even though it is not single msi-x vector. > > Actual scenario need some specific topology and server for overnight test. > > Issue can be seen on servers which has more than 16 logical CPUs and > > Thunderbolt series MR controller which supports at max 16 MSIx vectors. > > > > > > > > > > > Output of affinity hints. > > > > kernel version: > > > > Linux rhel7.3 4.15.0-rc1+ #2 SMP Mon Feb 5 12:13:34 EST 2018 > > > > x86_64 > > > > x86_64 > > > > x86_64 GNU/Linux > > > > PCI name is 83:00.0, dump its irq affinity: > > > > irq
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Tue, Feb 06, 2018 at 04:59:51PM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Ming Lei [mailto:ming@redhat.com] > > Sent: Tuesday, February 6, 2018 1:35 PM > > To: Kashyap Desai > > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar > Sandoval; > > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; > Peter > > Rivera; Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > Hi Kashyap, > > > > On Tue, Feb 06, 2018 at 11:33:50AM +0530, Kashyap Desai wrote: > > > > > We still have more than one reply queue ending up completion one > CPU. > > > > > > > > pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that means > > > > smp_affinity_enable has to be set as 1, but seems it is the default > > > setting. > > > > > > > > Please see kernel/irq/affinity.c, especially > > > > irq_calc_affinity_vectors() > > > which > > > > figures out an optimal number of vectors, and the computation is > > > > based > > > on > > > > cpumask_weight(cpu_possible_mask) now. If all offline CPUs are > > > > mapped to some of reply queues, these queues won't be active(no > > > > request submitted > > > to > > > > these queues). The mechanism of PCI_IRQ_AFFINITY basically makes > > > > sure > > > that > > > > more than one irq vector won't be handled by one same CPU, and the > > > > irq vector spread is done in irq_create_affinity_masks(). > > > > > > > > > Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver via > > > > > module parameter to simulate the issue. We need more number of > > > > > Online CPU than reply-queue. > > > > > > > > IMO, you don't need to simulate the issue, pci_alloc_irq_vectors( > > > > PCI_IRQ_AFFINITY) will handle that for you. You can dump the > > > > returned > > > irq > > > > vector number, num_possible_cpus()/num_online_cpus() and each irq > > > > vector's affinity assignment. > > > > > > > > > We may see completion redirected to original CPU because of > > > > > "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep one > > > > > CPU busy in local ISR routine. > > > > > > > > Could you dump each irq vector's affinity assignment of your > > > > megaraid in > > > your > > > > test? > > > > > > To quickly reproduce, I restricted to single MSI-x vector on > > > megaraid_sas driver. System has total 16 online CPUs. > > > > I suggest you don't do the restriction of single MSI-x vector, and just > use the > > device supported number of msi-x vector. > > Hi Ming, CPU lock up is seen even though it is not single msi-x vector. > Actual scenario need some specific topology and server for overnight test. > Issue can be seen on servers which has more than 16 logical CPUs and > Thunderbolt series MR controller which supports at max 16 MSIx vectors. > > > > > > > > Output of affinity hints. > > > kernel version: > > > Linux rhel7.3 4.15.0-rc1+ #2 SMP Mon Feb 5 12:13:34 EST 2018 x86_64 > > > x86_64 > > > x86_64 GNU/Linux > > > PCI name is 83:00.0, dump its irq affinity: > > > irq 105, cpu list 0-3,8-11 > > > > In this case, which CPU is selected for handling the interrupt is > decided by > > interrupt controller, and it is easy to cause CPU overload if interrupt > controller > > always selects one same CPU to handle the irq. > > > > > > > > Affinity mask is created properly, but only CPU-0 is overloaded with > > > interrupt processing. > > > > > > # numactl --hardware > > > available: 2 nodes (0-1) > > > node 0 cpus: 0 1 2 3 8 9 10 11 > > > node 0 size: 47861 MB > > > node 0 free: 46516 MB > > > node 1 cpus: 4 5 6 7 12 13 14 15 > > > node 1 size: 64491 MB > > > node 1 free: 62805 MB > > > node distances: > > > node 0 1 > > > 0: 10 21 > > > 1: 21 10 > > > > > > Output of system activities (sar). (gnice is 100% and it is consumed > > > in megaraid_sas ISR routine.) > > > > > > > > > 12:44:40 PM CP
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Ming Lei [mailto:ming@redhat.com] > Sent: Tuesday, February 6, 2018 1:35 PM > To: Kashyap Desai > Cc: Hannes Reinecke; Jens Axboe; linux-bl...@vger.kernel.org; Christoph > Hellwig; Mike Snitzer; linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; > Martin K . Petersen; James Bottomley; Christoph Hellwig; Don Brace; Peter > Rivera; Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > Hi Kashyap, > > On Tue, Feb 06, 2018 at 11:33:50AM +0530, Kashyap Desai wrote: > > > > We still have more than one reply queue ending up completion one CPU. > > > > > > pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that means > > > smp_affinity_enable has to be set as 1, but seems it is the default > > setting. > > > > > > Please see kernel/irq/affinity.c, especially > > > irq_calc_affinity_vectors() > > which > > > figures out an optimal number of vectors, and the computation is > > > based > > on > > > cpumask_weight(cpu_possible_mask) now. If all offline CPUs are > > > mapped to some of reply queues, these queues won't be active(no > > > request submitted > > to > > > these queues). The mechanism of PCI_IRQ_AFFINITY basically makes > > > sure > > that > > > more than one irq vector won't be handled by one same CPU, and the > > > irq vector spread is done in irq_create_affinity_masks(). > > > > > > > Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver via > > > > module parameter to simulate the issue. We need more number of > > > > Online CPU than reply-queue. > > > > > > IMO, you don't need to simulate the issue, pci_alloc_irq_vectors( > > > PCI_IRQ_AFFINITY) will handle that for you. You can dump the > > > returned > > irq > > > vector number, num_possible_cpus()/num_online_cpus() and each irq > > > vector's affinity assignment. > > > > > > > We may see completion redirected to original CPU because of > > > > "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep one > > > > CPU busy in local ISR routine. > > > > > > Could you dump each irq vector's affinity assignment of your > > > megaraid in > > your > > > test? > > > > To quickly reproduce, I restricted to single MSI-x vector on > > megaraid_sas driver. System has total 16 online CPUs. > > I suggest you don't do the restriction of single MSI-x vector, and just use the > device supported number of msi-x vector. Hi Ming, CPU lock up is seen even though it is not single msi-x vector. Actual scenario need some specific topology and server for overnight test. Issue can be seen on servers which has more than 16 logical CPUs and Thunderbolt series MR controller which supports at max 16 MSIx vectors. > > > > > Output of affinity hints. > > kernel version: > > Linux rhel7.3 4.15.0-rc1+ #2 SMP Mon Feb 5 12:13:34 EST 2018 x86_64 > > x86_64 > > x86_64 GNU/Linux > > PCI name is 83:00.0, dump its irq affinity: > > irq 105, cpu list 0-3,8-11 > > In this case, which CPU is selected for handling the interrupt is decided by > interrupt controller, and it is easy to cause CPU overload if interrupt controller > always selects one same CPU to handle the irq. > > > > > Affinity mask is created properly, but only CPU-0 is overloaded with > > interrupt processing. > > > > # numactl --hardware > > available: 2 nodes (0-1) > > node 0 cpus: 0 1 2 3 8 9 10 11 > > node 0 size: 47861 MB > > node 0 free: 46516 MB > > node 1 cpus: 4 5 6 7 12 13 14 15 > > node 1 size: 64491 MB > > node 1 free: 62805 MB > > node distances: > > node 0 1 > > 0: 10 21 > > 1: 21 10 > > > > Output of system activities (sar). (gnice is 100% and it is consumed > > in megaraid_sas ISR routine.) > > > > > > 12:44:40 PM CPU %usr %nice %sys %iowait%steal > > %irq %soft%guest%gnice %idle > > 12:44:41 PM all 6.03 0.0029.98 0.00 > > 0.00 0.000.000.000.00 63.99 > > 12:44:41 PM 0 0.00 0.00 0.000.00 > > 0.00 0.000.000.00 100.00 0 > > > > > > In my test, I used rq_affinity is set to 2. (QUEUE_FLAG_SAME_FORCE). I > > also used " host_tagset" V2 patch set for megaraid_sas. > > > > Us
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Tue, Feb 06, 2018 at 11:33:50AM +0530, Kashyap Desai wrote: > > > We still have more than one reply queue ending up completion one CPU. > > > > pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that means > > smp_affinity_enable has to be set as 1, but seems it is the default > setting. > > > > Please see kernel/irq/affinity.c, especially irq_calc_affinity_vectors() > which > > figures out an optimal number of vectors, and the computation is based > on > > cpumask_weight(cpu_possible_mask) now. If all offline CPUs are mapped to > > some of reply queues, these queues won't be active(no request submitted > to > > these queues). The mechanism of PCI_IRQ_AFFINITY basically makes sure > that > > more than one irq vector won't be handled by one same CPU, and the irq > > vector spread is done in irq_create_affinity_masks(). > > > > > Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver via > > > module parameter to simulate the issue. We need more number of Online > > > CPU than reply-queue. > > > > IMO, you don't need to simulate the issue, pci_alloc_irq_vectors( > > PCI_IRQ_AFFINITY) will handle that for you. You can dump the returned > irq > > vector number, num_possible_cpus()/num_online_cpus() and each irq > > vector's affinity assignment. > > > > > We may see completion redirected to original CPU because of > > > "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep one CPU > > > busy in local ISR routine. > > > > Could you dump each irq vector's affinity assignment of your megaraid in > your > > test? > > To quickly reproduce, I restricted to single MSI-x vector on megaraid_sas > driver. System has total 16 online CPUs. I suggest you don't do the restriction of single MSI-x vector, and just use the device supported number of msi-x vector. > > Output of affinity hints. > kernel version: > Linux rhel7.3 4.15.0-rc1+ #2 SMP Mon Feb 5 12:13:34 EST 2018 x86_64 x86_64 > x86_64 GNU/Linux > PCI name is 83:00.0, dump its irq affinity: > irq 105, cpu list 0-3,8-11 In this case, which CPU is selected for handling the interrupt is decided by interrupt controller, and it is easy to cause CPU overload if interrupt controller always selects one same CPU to handle the irq. > > Affinity mask is created properly, but only CPU-0 is overloaded with > interrupt processing. > > # numactl --hardware > available: 2 nodes (0-1) > node 0 cpus: 0 1 2 3 8 9 10 11 > node 0 size: 47861 MB > node 0 free: 46516 MB > node 1 cpus: 4 5 6 7 12 13 14 15 > node 1 size: 64491 MB > node 1 free: 62805 MB > node distances: > node 0 1 > 0: 10 21 > 1: 21 10 > > Output of system activities (sar). (gnice is 100% and it is consumed in > megaraid_sas ISR routine.) > > > 12:44:40 PM CPU %usr %nice %sys %iowait%steal > %irq %soft%guest%gnice %idle > 12:44:41 PM all 6.03 0.0029.98 0.00 > 0.00 0.000.000.000.00 63.99 > 12:44:41 PM 0 0.00 0.00 0.000.00 > 0.00 0.000.000.00 100.00 0 > > > In my test, I used rq_affinity is set to 2. (QUEUE_FLAG_SAME_FORCE). I > also used " host_tagset" V2 patch set for megaraid_sas. > > Using RFC requested in - > "https://marc.info/?l=linux-scsi=151601833418346=2 " lockup is avoided > (you can noticed that gnice is shifted to softirq. Even though it is 100% > consumed, There is always exit for existing completion loop due to > irqpoll_weight @irq_poll_init(). > > Average:CPU %usr %nice %sys %iowait%steal > %irq %soft%guest%gnice %idle > Average:all 4.25 0.0021.61 0.00 > 0.00 0.00 6.61 0.00 0.00 67.54 > Average: 0 0.00 0.00 0.00 0.00 > 0.00 0.00 100.000.00 0.00 0.00 > > > Hope this clarifies. We need different fix to avoid lockups. Can we > consider using irq poll interface if #CPU is more than Reply queue/MSI-x. > ? Please use the device's supported msi-x vectors number, and see if there is this issue. If there is, you can use irq poll too, which isn't contradictory with the blk-mq approach taken by this patchset. Hope I clarifies too, :-) Thanks, Ming
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> > We still have more than one reply queue ending up completion one CPU. > > pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that means > smp_affinity_enable has to be set as 1, but seems it is the default setting. > > Please see kernel/irq/affinity.c, especially irq_calc_affinity_vectors() which > figures out an optimal number of vectors, and the computation is based on > cpumask_weight(cpu_possible_mask) now. If all offline CPUs are mapped to > some of reply queues, these queues won't be active(no request submitted to > these queues). The mechanism of PCI_IRQ_AFFINITY basically makes sure that > more than one irq vector won't be handled by one same CPU, and the irq > vector spread is done in irq_create_affinity_masks(). > > > Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver via > > module parameter to simulate the issue. We need more number of Online > > CPU than reply-queue. > > IMO, you don't need to simulate the issue, pci_alloc_irq_vectors( > PCI_IRQ_AFFINITY) will handle that for you. You can dump the returned irq > vector number, num_possible_cpus()/num_online_cpus() and each irq > vector's affinity assignment. > > > We may see completion redirected to original CPU because of > > "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep one CPU > > busy in local ISR routine. > > Could you dump each irq vector's affinity assignment of your megaraid in your > test? To quickly reproduce, I restricted to single MSI-x vector on megaraid_sas driver. System has total 16 online CPUs. Output of affinity hints. kernel version: Linux rhel7.3 4.15.0-rc1+ #2 SMP Mon Feb 5 12:13:34 EST 2018 x86_64 x86_64 x86_64 GNU/Linux PCI name is 83:00.0, dump its irq affinity: irq 105, cpu list 0-3,8-11 Affinity mask is created properly, but only CPU-0 is overloaded with interrupt processing. # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 8 9 10 11 node 0 size: 47861 MB node 0 free: 46516 MB node 1 cpus: 4 5 6 7 12 13 14 15 node 1 size: 64491 MB node 1 free: 62805 MB node distances: node 0 1 0: 10 21 1: 21 10 Output of system activities (sar). (gnice is 100% and it is consumed in megaraid_sas ISR routine.) 12:44:40 PM CPU %usr %nice %sys %iowait%steal %irq %soft%guest%gnice %idle 12:44:41 PM all 6.03 0.0029.98 0.00 0.00 0.000.000.000.00 63.99 12:44:41 PM 0 0.00 0.00 0.000.00 0.00 0.000.000.00 100.00 0 In my test, I used rq_affinity is set to 2. (QUEUE_FLAG_SAME_FORCE). I also used " host_tagset" V2 patch set for megaraid_sas. Using RFC requested in - "https://marc.info/?l=linux-scsi=151601833418346=2 " lockup is avoided (you can noticed that gnice is shifted to softirq. Even though it is 100% consumed, There is always exit for existing completion loop due to irqpoll_weight @irq_poll_init(). Average:CPU %usr %nice %sys %iowait%steal %irq %soft%guest%gnice %idle Average:all 4.25 0.0021.61 0.00 0.00 0.00 6.61 0.00 0.00 67.54 Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 100.000.00 0.00 0.00 Hope this clarifies. We need different fix to avoid lockups. Can we consider using irq poll interface if #CPU is more than Reply queue/MSI-x. ? > > And the following script can do it easily, and the pci path (the 1st column of > lspci output) need to be passed, such as: 00:1c.4, > > #!/bin/sh > if [ $# -ge 1 ]; then > PCID=$1 > else > PCID=`lspci | grep "Non-Volatile memory" | cut -c1-7` fi PCIP=`find > /sys/devices -name *$PCID | grep pci` IRQS=`ls $PCIP/msi_irqs` > > echo "kernel version: " > uname -a > > echo "PCI name is $PCID, dump its irq affinity:" > for IRQ in $IRQS; do > CPUS=`cat /proc/irq/$IRQ/smp_affinity_list` > echo "\tirq $IRQ, cpu list $CPUS" > done > > > Thanks, > Ming
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On Mon, Feb 05, 2018 at 07:58:29AM +0100, Hannes Reinecke wrote: > On 02/03/2018 05:21 AM, Ming Lei wrote: > > Hi All, > > > > This patchset supports global tags which was started by Hannes originally: > > > > https://marc.info/?l=linux-block=149132580511346=2 > > > > Also inroduce 'force_blk_mq' to 'struct scsi_host_template', so that > > driver can avoid to support two IO paths(legacy and blk-mq), especially > > recent discusion mentioned that SCSI_MQ will be enabled at default soon. > > > > https://marc.info/?l=linux-scsi=151727684915589=2 > > > > With the above two changes, it should be easier to convert SCSI drivers' > > reply queue into blk-mq's hctx, then the automatic irq affinity issue can > > be solved easily, please see detailed descrption in commit log. > > > > Also drivers may require to complete request on the submission CPU > > for avoiding hard/soft deadlock, which can be done easily with blk_mq > > too. > > > > https://marc.info/?t=15160185141=1=2 > > > > The final patch uses the introduced 'force_blk_mq' to fix virtio_scsi > > so that IO hang issue can be avoided inside legacy IO path, this issue is > > a bit generic, at least HPSA/virtio-scsi are found broken with v4.15+. > > > > Thanks > > Ming > > > > Ming Lei (5): > > blk-mq: tags: define several fields of tags as pointer > > blk-mq: introduce BLK_MQ_F_GLOBAL_TAGS > > block: null_blk: introduce module parameter of 'g_global_tags' > > scsi: introduce force_blk_mq > > scsi: virtio_scsi: fix IO hang by irq vector automatic affinity > > > > block/bfq-iosched.c| 4 +-- > > block/blk-mq-debugfs.c | 11 > > block/blk-mq-sched.c | 2 +- > > block/blk-mq-tag.c | 67 > > +- > > block/blk-mq-tag.h | 15 --- > > block/blk-mq.c | 31 - > > block/blk-mq.h | 3 ++- > > block/kyber-iosched.c | 2 +- > > drivers/block/null_blk.c | 6 + > > drivers/scsi/hosts.c | 1 + > > drivers/scsi/virtio_scsi.c | 59 +++- > > include/linux/blk-mq.h | 2 ++ > > include/scsi/scsi_host.h | 3 +++ > > 13 files changed, 105 insertions(+), 101 deletions(-) > > > Thanks Ming for picking this up. > > I'll give it a shot and see how it behaves on other hardware. Hi Hannes, Thanks for looking at it. I am working on V2, which has fixed some issues, and added your patch of 'scsi: Add template flag 'host_tagset', but causes a HPSA kernel oops. Once it is fixed, I will posted V2 out, then there will be one real example about how to use global tags for converting reply queue to blk-mq hctx. https://github.com/ming1/linux/commits/v4.15-for-next-global-tags-V2 Thanks, Ming
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
Hi Kashyap, On Mon, Feb 05, 2018 at 12:35:13PM +0530, Kashyap Desai wrote: > > -Original Message- > > From: Hannes Reinecke [mailto:h...@suse.de] > > Sent: Monday, February 5, 2018 12:28 PM > > To: Ming Lei; Jens Axboe; linux-bl...@vger.kernel.org; Christoph Hellwig; > > Mike Snitzer > > Cc: linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; Martin K . > > Petersen; > > James Bottomley; Christoph Hellwig; Don Brace; Kashyap Desai; Peter > > Rivera; > > Paolo Bonzini; Laurence Oberman > > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > > force_blk_mq > > > > On 02/03/2018 05:21 AM, Ming Lei wrote: > > > Hi All, > > > > > > This patchset supports global tags which was started by Hannes > > > originally: > > > > > > https://marc.info/?l=linux-block=149132580511346=2 > > > > > > Also inroduce 'force_blk_mq' to 'struct scsi_host_template', so that > > > driver can avoid to support two IO paths(legacy and blk-mq), > > > especially recent discusion mentioned that SCSI_MQ will be enabled at > > default soon. > > > > > > https://marc.info/?l=linux-scsi=151727684915589=2 > > > > > > With the above two changes, it should be easier to convert SCSI drivers' > > > reply queue into blk-mq's hctx, then the automatic irq affinity issue > > > can be solved easily, please see detailed descrption in commit log. > > > > > > Also drivers may require to complete request on the submission CPU for > > > avoiding hard/soft deadlock, which can be done easily with blk_mq too. > > > > > > https://marc.info/?t=15160185141=1=2 > > > > > > The final patch uses the introduced 'force_blk_mq' to fix virtio_scsi > > > so that IO hang issue can be avoided inside legacy IO path, this issue > > > is a bit generic, at least HPSA/virtio-scsi are found broken with > > > v4.15+. > > > > > > Thanks > > > Ming > > > > > > Ming Lei (5): > > > blk-mq: tags: define several fields of tags as pointer > > > blk-mq: introduce BLK_MQ_F_GLOBAL_TAGS > > > block: null_blk: introduce module parameter of 'g_global_tags' > > > scsi: introduce force_blk_mq > > > scsi: virtio_scsi: fix IO hang by irq vector automatic affinity > > > > > > block/bfq-iosched.c| 4 +-- > > > block/blk-mq-debugfs.c | 11 > > > block/blk-mq-sched.c | 2 +- > > > block/blk-mq-tag.c | 67 > > > +- > > > block/blk-mq-tag.h | 15 --- > > > block/blk-mq.c | 31 - > > > block/blk-mq.h | 3 ++- > > > block/kyber-iosched.c | 2 +- > > > drivers/block/null_blk.c | 6 + > > > drivers/scsi/hosts.c | 1 + > > > drivers/scsi/virtio_scsi.c | 59 > > > +++- > > > include/linux/blk-mq.h | 2 ++ > > > include/scsi/scsi_host.h | 3 +++ > > > 13 files changed, 105 insertions(+), 101 deletions(-) > > > > > Thanks Ming for picking this up. > > > > I'll give it a shot and see how it behaves on other hardware. > > Ming - > > There is no way we can enable global tags from SCSI stack in this patch It has been done in V2 of this patchset, which will be posted out after HPSA's issue is fixed: https://github.com/ming1/linux/commits/v4.15-for-next-global-tags-V2 Global tags will be enabled easily via .host_tagset of scsi_host_template. > series. I still think we have no solution for issue described below in > this patch series. > https://marc.info/?t=15160185141=1=2 > > What we will be doing is just use global tag HBA wide instead of h/w queue > based. Right, that is just what the 1st three patches are doing. > We still have more than one reply queue ending up completion one CPU. pci_alloc_irq_vectors(PCI_IRQ_AFFINITY) has to be used, that means smp_affinity_enable has to be set as 1, but seems it is the default setting. Please see kernel/irq/affinity.c, especially irq_calc_affinity_vectors() which figures out an optimal number of vectors, and the computation is based on cpumask_weight(cpu_possible_mask) now. If all offline CPUs are mapped to some of reply queues, these queues won't be active(no request submitted to these queues). The mechanism of PCI_IRQ_AFFINITY basically makes sure that more than one irq vector won't be handled by one same CPU, and the irq vector spread is done in irq_create_affinity_mask
RE: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
> -Original Message- > From: Hannes Reinecke [mailto:h...@suse.de] > Sent: Monday, February 5, 2018 12:28 PM > To: Ming Lei; Jens Axboe; linux-bl...@vger.kernel.org; Christoph Hellwig; > Mike Snitzer > Cc: linux-scsi@vger.kernel.org; Arun Easi; Omar Sandoval; Martin K . > Petersen; > James Bottomley; Christoph Hellwig; Don Brace; Kashyap Desai; Peter > Rivera; > Paolo Bonzini; Laurence Oberman > Subject: Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce > force_blk_mq > > On 02/03/2018 05:21 AM, Ming Lei wrote: > > Hi All, > > > > This patchset supports global tags which was started by Hannes > > originally: > > > > https://marc.info/?l=linux-block=149132580511346=2 > > > > Also inroduce 'force_blk_mq' to 'struct scsi_host_template', so that > > driver can avoid to support two IO paths(legacy and blk-mq), > > especially recent discusion mentioned that SCSI_MQ will be enabled at > default soon. > > > > https://marc.info/?l=linux-scsi=151727684915589=2 > > > > With the above two changes, it should be easier to convert SCSI drivers' > > reply queue into blk-mq's hctx, then the automatic irq affinity issue > > can be solved easily, please see detailed descrption in commit log. > > > > Also drivers may require to complete request on the submission CPU for > > avoiding hard/soft deadlock, which can be done easily with blk_mq too. > > > > https://marc.info/?t=15160185141=1=2 > > > > The final patch uses the introduced 'force_blk_mq' to fix virtio_scsi > > so that IO hang issue can be avoided inside legacy IO path, this issue > > is a bit generic, at least HPSA/virtio-scsi are found broken with > > v4.15+. > > > > Thanks > > Ming > > > > Ming Lei (5): > > blk-mq: tags: define several fields of tags as pointer > > blk-mq: introduce BLK_MQ_F_GLOBAL_TAGS > > block: null_blk: introduce module parameter of 'g_global_tags' > > scsi: introduce force_blk_mq > > scsi: virtio_scsi: fix IO hang by irq vector automatic affinity > > > > block/bfq-iosched.c| 4 +-- > > block/blk-mq-debugfs.c | 11 > > block/blk-mq-sched.c | 2 +- > > block/blk-mq-tag.c | 67 > > +- > > block/blk-mq-tag.h | 15 --- > > block/blk-mq.c | 31 - > > block/blk-mq.h | 3 ++- > > block/kyber-iosched.c | 2 +- > > drivers/block/null_blk.c | 6 + > > drivers/scsi/hosts.c | 1 + > > drivers/scsi/virtio_scsi.c | 59 > > +++- > > include/linux/blk-mq.h | 2 ++ > > include/scsi/scsi_host.h | 3 +++ > > 13 files changed, 105 insertions(+), 101 deletions(-) > > > Thanks Ming for picking this up. > > I'll give it a shot and see how it behaves on other hardware. Ming - There is no way we can enable global tags from SCSI stack in this patch series. I still think we have no solution for issue described below in this patch series. https://marc.info/?t=15160185141=1=2 What we will be doing is just use global tag HBA wide instead of h/w queue based. We still have more than one reply queue ending up completion one CPU. Try to reduce MSI-x vector of megaraid_sas or mpt3sas driver via module parameter to simulate the issue. We need more number of Online CPU than reply-queue. We may see completion redirected to original CPU because of "QUEUE_FLAG_SAME_FORCE", but ISR of low level driver can keep one CPU busy in local ISR routine. Kashyap > > Cheers, > > Hannes > -- > Dr. Hannes Reinecke Teamlead Storage & Networking > h...@suse.de +49 911 74053 688 > SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg > GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 > (AG Nürnberg)
Re: [PATCH 0/5] blk-mq/scsi-mq: support global tags & introduce force_blk_mq
On 02/03/2018 05:21 AM, Ming Lei wrote: > Hi All, > > This patchset supports global tags which was started by Hannes originally: > > https://marc.info/?l=linux-block=149132580511346=2 > > Also inroduce 'force_blk_mq' to 'struct scsi_host_template', so that > driver can avoid to support two IO paths(legacy and blk-mq), especially > recent discusion mentioned that SCSI_MQ will be enabled at default soon. > > https://marc.info/?l=linux-scsi=151727684915589=2 > > With the above two changes, it should be easier to convert SCSI drivers' > reply queue into blk-mq's hctx, then the automatic irq affinity issue can > be solved easily, please see detailed descrption in commit log. > > Also drivers may require to complete request on the submission CPU > for avoiding hard/soft deadlock, which can be done easily with blk_mq > too. > > https://marc.info/?t=15160185141=1=2 > > The final patch uses the introduced 'force_blk_mq' to fix virtio_scsi > so that IO hang issue can be avoided inside legacy IO path, this issue is > a bit generic, at least HPSA/virtio-scsi are found broken with v4.15+. > > Thanks > Ming > > Ming Lei (5): > blk-mq: tags: define several fields of tags as pointer > blk-mq: introduce BLK_MQ_F_GLOBAL_TAGS > block: null_blk: introduce module parameter of 'g_global_tags' > scsi: introduce force_blk_mq > scsi: virtio_scsi: fix IO hang by irq vector automatic affinity > > block/bfq-iosched.c| 4 +-- > block/blk-mq-debugfs.c | 11 > block/blk-mq-sched.c | 2 +- > block/blk-mq-tag.c | 67 > +- > block/blk-mq-tag.h | 15 --- > block/blk-mq.c | 31 - > block/blk-mq.h | 3 ++- > block/kyber-iosched.c | 2 +- > drivers/block/null_blk.c | 6 + > drivers/scsi/hosts.c | 1 + > drivers/scsi/virtio_scsi.c | 59 +++- > include/linux/blk-mq.h | 2 ++ > include/scsi/scsi_host.h | 3 +++ > 13 files changed, 105 insertions(+), 101 deletions(-) > Thanks Ming for picking this up. I'll give it a shot and see how it behaves on other hardware. Cheers, Hannes -- Dr. Hannes ReineckeTeamlead Storage & Networking h...@suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
