Switching to MQ by default may generate some bug reports

Hi Christoph, I know the reasons for switching to MQ by default but just be aware that it's not without hazards albeit it the biggest issues I've seen are switching CFQ to BFQ. On my home grid, there is some experimental automatic testing running every few weeks searching for regressions.

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 03, 2017 at 05:17:21PM +0800, Ming Lei wrote: > Hi Mel Gorman, > > On Thu, Aug 3, 2017 at 4:51 PM, Mel Gorman > wrote: > > Hi Christoph, > > > > I know the reasons for switching to MQ by default but just be aware that > > it's > > not without hazards

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 03, 2017 at 05:57:50PM +0800, Ming Lei wrote: > On Thu, Aug 3, 2017 at 5:42 PM, Mel Gorman > wrote: > > On Thu, Aug 03, 2017 at 05:17:21PM +0800, Ming Lei wrote: > >> Hi Mel Gorman, > >> > >> On Thu, Aug 3, 2017 at 4:51 PM, Mel Gorman

Re: Switching to MQ by default may generate some bug reports

Hi Mel Gorman, On Thu, Aug 3, 2017 at 4:51 PM, Mel Gorman wrote: > Hi Christoph, > > I know the reasons for switching to MQ by default but just be aware that it's > not without hazards albeit it the biggest issues I've seen are switching > CFQ to BFQ. On my home

Re: Switching to MQ by default may generate some bug reports

> Il giorno 03 ago 2017, alle ore 10:51, Mel Gorman > ha scritto: > > Hi Christoph, > > I know the reasons for switching to MQ by default but just be aware that it's > not without hazards albeit it the biggest issues I've seen are switching > CFQ to BFQ. On my

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 3, 2017 at 5:42 PM, Mel Gorman wrote: > On Thu, Aug 03, 2017 at 05:17:21PM +0800, Ming Lei wrote: >> Hi Mel Gorman, >> >> On Thu, Aug 3, 2017 at 4:51 PM, Mel Gorman >> wrote: >> > Hi Christoph, >> > >> > I know the reasons

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 03, 2017 at 11:44:06AM +0200, Paolo Valente wrote: > > That series is dealing with problems with legacy-deadline vs mq-none where > > as the bulk of the problems reported in this mail are related to > > legacy-CFQ vs mq-BFQ. > > > > Out-of-curiosity: you get no regression with

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 03, 2017 at 11:21:59AM +0200, Paolo Valente wrote: > > For Paulo, if you want to try preemptively dealing with regression reports > > before 4.13 releases then all the tests in question can be reproduced with > > https://github.com/gormanm/mmtests . The most relevant test

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 3, 2017 at 5:17 PM, Ming Lei wrote: > Hi Mel Gorman, > > On Thu, Aug 3, 2017 at 4:51 PM, Mel Gorman > wrote: >> Hi Christoph, >> >> I know the reasons for switching to MQ by default but just be aware that it's >> not without

Re: [PATCH 17/19] bcache: fix for gc and write-back race

On 2017/7/1 上午4:43, bca...@lists.ewheeler.net wrote: > From: Tang Junhui > > gc and write-back get raced (see the email "bcache get stucked" I sended > before): > gc thread write-back thread > |

[PATCH] bio-integrity: revert "stop abusing bi_end_io"

On Wed, 2 Aug 2017, Christoph Hellwig wrote: > On Wed, Aug 02, 2017 at 02:27:50PM +0200, Milan Broz wrote: > > In dm-integrity target we register integrity profile that have > > both generate_fn and verify_fn callbacks set to NULL. > > > > This is used if dm-integrity is stacked under a

Re: [PATCH BUGFIX/IMPROVEMENT] block, bfq: improve and refactor throughput-boosting logic

On 08/03/2017 10:48 AM, Paolo Valente wrote: > When a queue associated with a process remains empty, there are cases > where throughput gets boosted if the device is idled to await the > arrival of a new I/O request for that queue. Currently, BFQ assumes > that one of these cases is when the

[PATCH BUGFIX/IMPROVEMENT] block, bfq: improve and refactor throughput-boosting logic

When a queue associated with a process remains empty, there are cases where throughput gets boosted if the device is idled to await the arrival of a new I/O request for that queue. Currently, BFQ assumes that one of these cases is when the device has no internal queueing (regardless of the

Re: [PATCH 04/14] blk-mq-sched: improve dispatching from sw queue

On Thu, 2017-08-03 at 11:13 +0800, Ming Lei wrote: > On Thu, Aug 03, 2017 at 01:35:29AM +, Bart Van Assche wrote: > > On Wed, 2017-08-02 at 11:31 +0800, Ming Lei wrote: > > > On Tue, Aug 01, 2017 at 03:11:42PM +, Bart Van Assche wrote: > > > > On Tue, 2017-08-01 at 18:50 +0800, Ming Lei

[PATCH 2/4] block: pass in queue to inflight accounting

No functional change in this patch, just in preparation for basing the inflight mechanism on the queue in question. Signed-off-by: Jens Axboe --- block/bio.c | 16 block/blk-core.c | 22 -- block/blk-merge.c

[PATCH 3/4] blk-mq: provide internal in-flight variant

We don't have to inc/dec some counter, since we can just iterate the tags. That makes inc/dec a noop, but means we have to iterate busy tags to get an in-flight count. Signed-off-by: Jens Axboe --- block/blk-mq.c| 24 block/blk-mq.h| 2

[PATCH 0/4] block: more scalable inflight tracking

One scaling issue we currently have in the block code is the inflight accounting. It's based on a per-device atomic count for reads and writes, which means that even for an mq device with lots of hardware queues, we end up dirtying a per-device cacheline for each IO. The issue can easily be

[PATCH 1/4] blk-mq-tag: check for NULL rq when iterating tags

Since we introduced blk-mq-sched, the tags->rqs[] array has been dynamically assigned. So we need to check for NULL when iterating, since we could be racing with completion. This is perfectly safe, since the memory backing of the request is never going away while the device is alive. Only the

Re: [PATCH 3/4] blk-mq: provide internal in-flight variant

On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: > We don't have to inc/dec some counter, since we can just > iterate the tags. That makes inc/dec a noop, but means we > have to iterate busy tags to get an in-flight count. > [ ... ] > +unsigned int blk_mq_in_flight(struct request_queue *q, > +

Re: [PATCH 3/4] blk-mq: provide internal in-flight variant

On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: > +static void blk_mq_check_inflight(struct blk_mq_hw_ctx *hctx, > + struct request *rq, void *priv, > + bool reserved) > +{ > + struct mq_inflight *mi = priv; > + > + if

Re: [PATCH 1/4] blk-mq-tag: check for NULL rq when iterating tags

On 08/03/2017 02:29 PM, Bart Van Assche wrote: > On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: >> Since we introduced blk-mq-sched, the tags->rqs[] array has been >> dynamically assigned. So we need to check for NULL when iterating, >> since we could be racing with completion. >> >> This is

Re: [PATCH 1/4] blk-mq-tag: check for NULL rq when iterating tags

On 08/03/2017 02:35 PM, Jens Axboe wrote: >> I agree with what you wrote in the description of this patch. >> However, since I have not yet found the code that clears tags->rqs[], >> would it be possible to show me that code? > > Since it's been a month since I wrote this code, I went and looked

Re: [PATCH 2/4] block: pass in queue to inflight accounting

On 08/03/2017 02:35 PM, Bart Van Assche wrote: > On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: >> No functional change in this patch, just in preparation for >> basing the inflight mechanism on the queue in question. >> >> Signed-off-by: Jens Axboe >> --- >> block/bio.c

[PATCH 4/4] blk-mq: enable checking two part inflight counts at the same time

We only need to iterate the queues and tags once, we if just pass in both partition structs. Add part_in_flight_double() for returning this value. Signed-off-by: Jens Axboe --- block/blk-core.c | 34 +++--- block/blk-mq.c| 25

Re: [PATCH 1/4] blk-mq-tag: check for NULL rq when iterating tags

On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: > Since we introduced blk-mq-sched, the tags->rqs[] array has been > dynamically assigned. So we need to check for NULL when iterating, > since we could be racing with completion. > > This is perfectly safe, since the memory backing of the

Re: [PATCH 2/4] block: pass in queue to inflight accounting

On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: > No functional change in this patch, just in preparation for > basing the inflight mechanism on the queue in question. > > Signed-off-by: Jens Axboe > --- > block/bio.c | 16 >

Re: [PATCH 1/4] blk-mq-tag: check for NULL rq when iterating tags

On 08/03/2017 02:50 PM, Bart Van Assche wrote: > On Thu, 2017-08-03 at 14:40 -0600, Jens Axboe wrote: >> On 08/03/2017 02:35 PM, Jens Axboe wrote: I agree with what you wrote in the description of this patch. However, since I have not yet found the code that clears tags->rqs[],

Re: [PATCH 4/4] blk-mq: enable checking two part inflight counts at the same time

On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: > static void blk_mq_check_inflight(struct blk_mq_hw_ctx *hctx, > @@ -97,17 +98,25 @@ static void blk_mq_check_inflight(struct blk_mq_hw_ctx > *hctx, > { > struct mq_inflight *mi = priv; > > - if (rq->part == mi->part) > -

Re: [PATCH 3/4] blk-mq: provide internal in-flight variant

On 08/03/2017 02:41 PM, Bart Van Assche wrote: > On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: >> We don't have to inc/dec some counter, since we can just >> iterate the tags. That makes inc/dec a noop, but means we >> have to iterate busy tags to get an in-flight count. >> [ ... ] >>

Re: [PATCH 1/4] blk-mq-tag: check for NULL rq when iterating tags

On Thu, 2017-08-03 at 14:40 -0600, Jens Axboe wrote: > On 08/03/2017 02:35 PM, Jens Axboe wrote: > > > I agree with what you wrote in the description of this patch. > > > However, since I have not yet found the code that clears tags->rqs[], > > > would it be possible to show me that code? > > > >

Re: [PATCH 3/4] blk-mq: provide internal in-flight variant

On Thu, 2017-08-03 at 14:45 -0600, Jens Axboe wrote: > The inc/dec goes away for mq, the non-mq path still has to use them. I just > move them as well. Could be a prep patch, but it's just moving the code out > of the header and into a normal C file instead. Hello Jens, I misread that part of

Re: [PATCH 4/4] blk-mq: enable checking two part inflight counts at the same time

On 08/03/2017 03:29 PM, Bart Van Assche wrote: > On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: >> static void blk_mq_check_inflight(struct blk_mq_hw_ctx *hctx, >> @@ -97,17 +98,25 @@ static void blk_mq_check_inflight(struct blk_mq_hw_ctx >> *hctx, >> { >> struct mq_inflight *mi =

Re: [PATCH 4/4] blk-mq: enable checking two part inflight counts at the same time

On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: > We only need to iterate the queues and tags once, we if just pass Hello Jens, Regarding the patch description: it seems to me like in the text "we if" these two words appear in the wrong order? Bart.

Re: [PATCH 3/4] blk-mq: provide internal in-flight variant

On 08/03/2017 03:25 PM, Bart Van Assche wrote: > On Thu, 2017-08-03 at 14:01 -0600, Jens Axboe wrote: >> +static void blk_mq_check_inflight(struct blk_mq_hw_ctx *hctx, >> + struct request *rq, void *priv, >> + bool reserved) >> +{ >> +

[PATCH BUGFIX/IMPROVEMENT V2 1/2] block,bfq: refactor device-idling logic

The logic that decides whether to idle the device is scattered across three functions. Almost all of the logic is in the function bfq_bfqq_may_idle, but (1) part of the decision is made in bfq_update_idle_window, and (2) the function bfq_bfqq_must_idle may switch off idling regardless of the

[PATCH BUGFIX/IMPROVEMENT V2 0/2] block, bfq: improve and refactor throughput-boosting logic

Hi, these two patches improve throughput-boosting logic in two aspects. The first patch refactors the parts of the device-idling logic, related to throughput boosting, that are still scattered across the source file bfq-iosched.c. The patch concetrates all the logic in one function. The second

[PATCH BUGFIX/IMPROVEMENT V2 2/2] block, bfq: boost throughput with flash-based non-queueing devices

When a queue associated with a process remains empty, there are cases where throughput gets boosted if the device is idled to await the arrival of a new I/O request for that queue. Currently, BFQ assumes that one of these cases is when the device has no internal queueing (regardless of the

Re: Switching to MQ by default may generate some bug reports

On Thu, Aug 3, 2017 at 6:47 PM, Mel Gorman wrote: > On Thu, Aug 03, 2017 at 05:57:50PM +0800, Ming Lei wrote: >> On Thu, Aug 3, 2017 at 5:42 PM, Mel Gorman >> wrote: >> > On Thu, Aug 03, 2017 at 05:17:21PM +0800, Ming Lei wrote: >> >> Hi