Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Mike Snitzer]

On Mon, Nov 20 2017 at  4:23pm -0500,
Shaohua Li  wrote:

> On Mon, Nov 20, 2017 at 04:02:03PM -0500, Mike Snitzer wrote:
> > On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi  wrote:
> > > From: Joseph Qi 
> > >
> > > dm device set QUEUE_FLAG_NONROT in resume, which is after register
> > > queue. That is to mean, the previous initialization in
> > > blk_throtl_register_queue is wrong in this case.
> > > Fix it by checking and then updating the info during root tg
> > > initialization as we don't have a better choice.
> > 
> > Given DM motivated this change, curious why you didn't send this to 
> > dm-devel?
> > 
> > In any case, not sure why you need to reference "resume".  Very few
> > people will appreciate that detail.
> > 
> > Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
> > is registered.
> > 
> > > Signed-off-by: Joseph Qi 
> > > ---
> > >  block/blk-throttle.c | 17 +
> > >  1 file changed, 17 insertions(+)
> > >
> > > diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> > > index bf52035..6d6b220 100644
> > > --- a/block/blk-throttle.c
> > > +++ b/block/blk-throttle.c
> > > @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data 
> > > *pd)
> > > if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
> > > sq->parent_sq = _to_tg(blkg->parent)->service_queue;
> > > tg->td = td;
> > > +
> > > +   /*
> > > +* dm device set QUEUE_FLAG_NONROT in resume, which is after 
> > > resister
> > > +* queue, so the previous initialization is wrong in this case. 
> > > Check
> > > +* and update it here.
> > > +*/
> > 
> > typo: s/resister/register/
> > But again, best to say:
> > DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...
> > 
> > Also, an alternative to your patch would be to have DM's
> > dm_table_set_restrictions() call a blk-throttle function to setup
> > blk-throttle after it is done setting queue flags?
> > Saves all other drivers from having to run this 2 stage initialization
> > code.. just a thought.
> 
> Though the patch is a workaround, I'd prefer limiting the code into
> blk-throttle itself. Populating it to drivers is a layer violation to me. The
> initialization only runs once, so I think it's ok.

I fail to see how this is a "workaround".  Whatever you call it, the
blk-throttle code needs to cope with the fact that DM's request_queue
setup is different than other block devices.

If having DM call a function that blk-throttle provides (code that is in
block/blk-throttle.c) I fail to see the layer violation.  No more a
layer violation than any of the other block core code that block drivers
call to setup the request_queue's capabilities.

But I'm not well-versed on blk-throttle code (hence DM coming up lacking
relative to it).  SO I'll defer to you guys to sort it out how you think
best.

Thanks,
Mike

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Joseph Qi]

On 17/11/21 01:15, Shaohua Li wrote:
> On Mon, Nov 20, 2017 at 10:00:27AM +0800, Joseph Qi wrote:
>> From: Joseph Qi 
>>
>> dm device set QUEUE_FLAG_NONROT in resume, which is after register
>> queue. That is to mean, the previous initialization in
>> blk_throtl_register_queue is wrong in this case.
>> Fix it by checking and then updating the info during root tg
>> initialization as we don't have a better choice.
>>
>> Signed-off-by: Joseph Qi 
>> ---
>>  block/blk-throttle.c | 17 +
>>  1 file changed, 17 insertions(+)
>>
>> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
>> index bf52035..6d6b220 100644
>> --- a/block/blk-throttle.c
>> +++ b/block/blk-throttle.c
>> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>>  if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>>  sq->parent_sq = _to_tg(blkg->parent)->service_queue;
>>  tg->td = td;
>> +
>> +/*
>> + * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
>> + * queue, so the previous initialization is wrong in this case. Check
>> + * and update it here.
>> + */
>> +if (blk_queue_nonrot(blkg->q) &&
>> +td->filtered_latency != LATENCY_FILTERED_SSD) {
>> +int i;
>> +
>> +td->throtl_slice = DFL_THROTL_SLICE_SSD;
> 
> if CONFIG_BLK_DEV_THROTTLING_LOW isn't not set, we use old slice, can you do
> the same thing here? Otherwise,
> Reviewed-by: Shaohua Li 
> 
Sure, I will update and resend it. Thanks for pointing this out.

Thanks,
Joseph

>> +td->filtered_latency = LATENCY_FILTERED_SSD;
>> +for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
>> +td->avg_buckets[READ][i].latency = 0;
>> +td->avg_buckets[WRITE][i].latency = 0;
>> +}
>> +}
>>  }
>>  
>>  /*
>> -- 
>> 1.9.4

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Joseph Qi]

Hi Mike,

On 17/11/21 05:02, Mike Snitzer wrote:
> On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi  wrote:
>> From: Joseph Qi 
>>
>> dm device set QUEUE_FLAG_NONROT in resume, which is after register
>> queue. That is to mean, the previous initialization in
>> blk_throtl_register_queue is wrong in this case.
>> Fix it by checking and then updating the info during root tg
>> initialization as we don't have a better choice.
> 
> Given DM motivated this change, curious why you didn't send this to dm-devel?
> 
Since this is io.low specifically, I don't want the change in device
mapper layer. But I do agree with you that I have to cc dm-devel as
well. Sorry for missing that.

> In any case, not sure why you need to reference "resume".  Very few
> people will appreciate that detail.
> 
> Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
> is registered.
> 
Thanks for your advice. I will update and resend it.

Thanks,
Joseph

>> Signed-off-by: Joseph Qi 
>> ---
>>  block/blk-throttle.c | 17 +
>>  1 file changed, 17 insertions(+)
>>
>> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
>> index bf52035..6d6b220 100644
>> --- a/block/blk-throttle.c
>> +++ b/block/blk-throttle.c
>> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>> if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>> sq->parent_sq = _to_tg(blkg->parent)->service_queue;
>> tg->td = td;
>> +
>> +   /*
>> +* dm device set QUEUE_FLAG_NONROT in resume, which is after resister
>> +* queue, so the previous initialization is wrong in this case. Check
>> +* and update it here.
>> +*/
> 
> typo: s/resister/register/
> But again, best to say:
> DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...
> 
> Also, an alternative to your patch would be to have DM's
> dm_table_set_restrictions() call a blk-throttle function to setup
> blk-throttle after it is done setting queue flags?
> Saves all other drivers from having to run this 2 stage initialization
> code.. just a thought.
> 
> Mike
> 

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Shaohua Li]

On Mon, Nov 20, 2017 at 04:02:03PM -0500, Mike Snitzer wrote:
> On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi  wrote:
> > From: Joseph Qi 
> >
> > dm device set QUEUE_FLAG_NONROT in resume, which is after register
> > queue. That is to mean, the previous initialization in
> > blk_throtl_register_queue is wrong in this case.
> > Fix it by checking and then updating the info during root tg
> > initialization as we don't have a better choice.
> 
> Given DM motivated this change, curious why you didn't send this to dm-devel?
> 
> In any case, not sure why you need to reference "resume".  Very few
> people will appreciate that detail.
> 
> Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
> is registered.
> 
> > Signed-off-by: Joseph Qi 
> > ---
> >  block/blk-throttle.c | 17 +
> >  1 file changed, 17 insertions(+)
> >
> > diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> > index bf52035..6d6b220 100644
> > --- a/block/blk-throttle.c
> > +++ b/block/blk-throttle.c
> > @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
> > if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
> > sq->parent_sq = _to_tg(blkg->parent)->service_queue;
> > tg->td = td;
> > +
> > +   /*
> > +* dm device set QUEUE_FLAG_NONROT in resume, which is after 
> > resister
> > +* queue, so the previous initialization is wrong in this case. 
> > Check
> > +* and update it here.
> > +*/
> 
> typo: s/resister/register/
> But again, best to say:
> DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...
> 
> Also, an alternative to your patch would be to have DM's
> dm_table_set_restrictions() call a blk-throttle function to setup
> blk-throttle after it is done setting queue flags?
> Saves all other drivers from having to run this 2 stage initialization
> code.. just a thought.

Though the patch is a workaround, I'd prefer limiting the code into
blk-throttle itself. Populating it to drivers is a layer violation to me. The
initialization only runs once, so I think it's ok.

Thanks,
Shaohua

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Mike Snitzer]

On Sun, Nov 19, 2017 at 9:00 PM, Joseph Qi  wrote:
> From: Joseph Qi 
>
> dm device set QUEUE_FLAG_NONROT in resume, which is after register
> queue. That is to mean, the previous initialization in
> blk_throtl_register_queue is wrong in this case.
> Fix it by checking and then updating the info during root tg
> initialization as we don't have a better choice.

Given DM motivated this change, curious why you didn't send this to dm-devel?

In any case, not sure why you need to reference "resume".  Very few
people will appreciate that detail.

Better to just say: DM device sets QUEUE_FLAG_NONROT after the queue
is registered.

> Signed-off-by: Joseph Qi 
> ---
>  block/blk-throttle.c | 17 +
>  1 file changed, 17 insertions(+)
>
> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> index bf52035..6d6b220 100644
> --- a/block/blk-throttle.c
> +++ b/block/blk-throttle.c
> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
> if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
> sq->parent_sq = _to_tg(blkg->parent)->service_queue;
> tg->td = td;
> +
> +   /*
> +* dm device set QUEUE_FLAG_NONROT in resume, which is after resister
> +* queue, so the previous initialization is wrong in this case. Check
> +* and update it here.
> +*/

typo: s/resister/register/
But again, best to say:
DM device sets QUEUE_FLAG_NONROT after the queue is registered, ...

Also, an alternative to your patch would be to have DM's
dm_table_set_restrictions() call a blk-throttle function to setup
blk-throttle after it is done setting queue flags?
Saves all other drivers from having to run this 2 stage initialization
code.. just a thought.

Mike

Re: [PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Shaohua Li]

On Mon, Nov 20, 2017 at 10:00:27AM +0800, Joseph Qi wrote:
> From: Joseph Qi 
> 
> dm device set QUEUE_FLAG_NONROT in resume, which is after register
> queue. That is to mean, the previous initialization in
> blk_throtl_register_queue is wrong in this case.
> Fix it by checking and then updating the info during root tg
> initialization as we don't have a better choice.
> 
> Signed-off-by: Joseph Qi 
> ---
>  block/blk-throttle.c | 17 +
>  1 file changed, 17 insertions(+)
> 
> diff --git a/block/blk-throttle.c b/block/blk-throttle.c
> index bf52035..6d6b220 100644
> --- a/block/blk-throttle.c
> +++ b/block/blk-throttle.c
> @@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
>   if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
>   sq->parent_sq = _to_tg(blkg->parent)->service_queue;
>   tg->td = td;
> +
> + /*
> +  * dm device set QUEUE_FLAG_NONROT in resume, which is after resister
> +  * queue, so the previous initialization is wrong in this case. Check
> +  * and update it here.
> +  */
> + if (blk_queue_nonrot(blkg->q) &&
> + td->filtered_latency != LATENCY_FILTERED_SSD) {
> + int i;
> +
> + td->throtl_slice = DFL_THROTL_SLICE_SSD;

if CONFIG_BLK_DEV_THROTTLING_LOW isn't not set, we use old slice, can you do
the same thing here? Otherwise,
Reviewed-by: Shaohua Li 

> + td->filtered_latency = LATENCY_FILTERED_SSD;
> + for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
> + td->avg_buckets[READ][i].latency = 0;
> + td->avg_buckets[WRITE][i].latency = 0;
> + }
> + }
>  }
>  
>  /*
> -- 
> 1.9.4

[PATCH v2 2/2] blk-throttle: fix wrong initialization in case of dm device

[Joseph Qi]

From: Joseph Qi 

dm device set QUEUE_FLAG_NONROT in resume, which is after register
queue. That is to mean, the previous initialization in
blk_throtl_register_queue is wrong in this case.
Fix it by checking and then updating the info during root tg
initialization as we don't have a better choice.

Signed-off-by: Joseph Qi 
---
 block/blk-throttle.c | 17 +
 1 file changed, 17 insertions(+)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index bf52035..6d6b220 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -541,6 +541,23 @@ static void throtl_pd_init(struct blkg_policy_data *pd)
if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
sq->parent_sq = _to_tg(blkg->parent)->service_queue;
tg->td = td;
+
+   /*
+* dm device set QUEUE_FLAG_NONROT in resume, which is after resister
+* queue, so the previous initialization is wrong in this case. Check
+* and update it here.
+*/
+   if (blk_queue_nonrot(blkg->q) &&
+   td->filtered_latency != LATENCY_FILTERED_SSD) {
+   int i;
+
+   td->throtl_slice = DFL_THROTL_SLICE_SSD;
+   td->filtered_latency = LATENCY_FILTERED_SSD;
+   for (i = 0; i < LATENCY_BUCKET_SIZE; i++) {
+   td->avg_buckets[READ][i].latency = 0;
+   td->avg_buckets[WRITE][i].latency = 0;
+   }
+   }
 }
 
 /*
-- 
1.9.4