From: Pavel Begunkov <[email protected]> Commit 4bc6339a583cec650b05 ("block: move blk_stat_add() to __blk_mq_end_request()") moved blk_stat_add() to reuse ktime_get_ns(), so now it's called after blk_update_request(), which zeroes rq->__data_len. Without length, blk_stat_add() can't calculate stat bucket and returns error, effectively disabling hybrid polling.
v2: Hybrid polling needs pure io time for precision, but according to the feedback from Omar Sandoval other components require end-to-end time. So, it can't be reused, and should be sampled twice insted. Signed-off-by: Pavel Begunkov <[email protected]> --- block/blk-mq.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 32b8ad3d341b..907799282d57 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -537,11 +537,6 @@ inline void __blk_mq_end_request(struct request *rq, blk_status_t error) if (blk_mq_need_time_stamp(rq)) now = ktime_get_ns(); - if (rq->rq_flags & RQF_STATS) { - blk_mq_poll_stats_start(rq->q); - blk_stat_add(rq, now); - } - if (rq->internal_tag != -1) blk_mq_sched_completed_request(rq, now); @@ -580,6 +575,11 @@ static void __blk_mq_complete_request(struct request *rq) int cpu; WRITE_ONCE(rq->state, MQ_RQ_COMPLETE); + + if (rq->rq_flags & RQF_STATS) { + blk_mq_poll_stats_start(rq->q); + blk_stat_add(rq, ktime_get_ns()); + } /* * Most of single queue controllers, there is only one irq vector * for handling IO completion, and the only irq's affinity is set -- 2.21.0
