Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Joel Fernandes (Google)]

On Thu, Jul 27, 2017 at 12:14 AM, Viresh Kumar  wrote:
> On 26-07-17, 23:13, Joel Fernandes (Google) wrote:
>> On Wed, Jul 26, 2017 at 10:50 PM, Viresh Kumar  
>> wrote:
>> > On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
>> >> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  
>> >> wrote:
>> >> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct 
>> >> > update_util_data *hook, u64 time,
>> >> > sugov_set_iowait_boost(sg_cpu, time, flags);
>> >> > sg_cpu->last_update = time;
>> >> >
>> >> > -   if (!sugov_should_update_freq(sg_policy, time))
>> >> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
>> >> > return;
>> >>
>> >> Since with the remote callbacks now possible, isn't it unsafe to
>> >> modify sg_cpu and sg_policy structures without a lock in
>> >> sugov_update_single?
>> >>
>> >> Unlike sugov_update_shared, we don't acquire any lock in
>> >> sugov_update_single before updating these structures. Did I miss
>> >> something?
>> >
>> > As Peter already mentioned it earlier, the callbacks are called with
>> > rq locks held and so sugov_update_single() wouldn't get called in
>> > parallel for a target CPU.
>>
>> Ah ok, I have to catch up with that discussion since I missed the
>> whole thing. Now that you will have me on CC, that shouldn't happen,
>> thanks and sorry about the noise.
>>
>> > That's the only race you were worried about ?
>>
>> Yes. So then in that case, makes sense to move raw_spin_lock in
>> sugov_update_shared further down? (Just discussing, this point is
>> independent of your patch), Something like:
>
> Even that was discussed tomorrow with Peter :)
>
> No it wouldn't work because sg_cpu->util we are updating here may be
> getting read from some other cpu that shares policy with sg_cpu.
>

Ok. yes you are right :) thank you Viresh and Peter for the clarification.

thanks,

-Joel

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Joel Fernandes (Google)]

On Thu, Jul 27, 2017 at 12:14 AM, Viresh Kumar  wrote:
> On 26-07-17, 23:13, Joel Fernandes (Google) wrote:
>> On Wed, Jul 26, 2017 at 10:50 PM, Viresh Kumar  
>> wrote:
>> > On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
>> >> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  
>> >> wrote:
>> >> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct 
>> >> > update_util_data *hook, u64 time,
>> >> > sugov_set_iowait_boost(sg_cpu, time, flags);
>> >> > sg_cpu->last_update = time;
>> >> >
>> >> > -   if (!sugov_should_update_freq(sg_policy, time))
>> >> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
>> >> > return;
>> >>
>> >> Since with the remote callbacks now possible, isn't it unsafe to
>> >> modify sg_cpu and sg_policy structures without a lock in
>> >> sugov_update_single?
>> >>
>> >> Unlike sugov_update_shared, we don't acquire any lock in
>> >> sugov_update_single before updating these structures. Did I miss
>> >> something?
>> >
>> > As Peter already mentioned it earlier, the callbacks are called with
>> > rq locks held and so sugov_update_single() wouldn't get called in
>> > parallel for a target CPU.
>>
>> Ah ok, I have to catch up with that discussion since I missed the
>> whole thing. Now that you will have me on CC, that shouldn't happen,
>> thanks and sorry about the noise.
>>
>> > That's the only race you were worried about ?
>>
>> Yes. So then in that case, makes sense to move raw_spin_lock in
>> sugov_update_shared further down? (Just discussing, this point is
>> independent of your patch), Something like:
>
> Even that was discussed tomorrow with Peter :)
>
> No it wouldn't work because sg_cpu->util we are updating here may be
> getting read from some other cpu that shares policy with sg_cpu.
>

Ok. yes you are right :) thank you Viresh and Peter for the clarification.

thanks,

-Joel

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Peter Zijlstra]

On Thu, Jul 27, 2017 at 12:44:41PM +0530, Viresh Kumar wrote:
> Even that was discussed tomorrow with Peter :)

Just to clarify I don't have a time machine. That discussion was
_yesterday_,... I think :-)

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Peter Zijlstra]

On Thu, Jul 27, 2017 at 12:44:41PM +0530, Viresh Kumar wrote:
> Even that was discussed tomorrow with Peter :)

Just to clarify I don't have a time machine. That discussion was
_yesterday_,... I think :-)

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Viresh Kumar]

On 26-07-17, 23:13, Joel Fernandes (Google) wrote:
> On Wed, Jul 26, 2017 at 10:50 PM, Viresh Kumar  
> wrote:
> > On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
> >> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  
> >> wrote:
> >> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct 
> >> > update_util_data *hook, u64 time,
> >> > sugov_set_iowait_boost(sg_cpu, time, flags);
> >> > sg_cpu->last_update = time;
> >> >
> >> > -   if (!sugov_should_update_freq(sg_policy, time))
> >> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
> >> > return;
> >>
> >> Since with the remote callbacks now possible, isn't it unsafe to
> >> modify sg_cpu and sg_policy structures without a lock in
> >> sugov_update_single?
> >>
> >> Unlike sugov_update_shared, we don't acquire any lock in
> >> sugov_update_single before updating these structures. Did I miss
> >> something?
> >
> > As Peter already mentioned it earlier, the callbacks are called with
> > rq locks held and so sugov_update_single() wouldn't get called in
> > parallel for a target CPU.
> 
> Ah ok, I have to catch up with that discussion since I missed the
> whole thing. Now that you will have me on CC, that shouldn't happen,
> thanks and sorry about the noise.
> 
> > That's the only race you were worried about ?
> 
> Yes. So then in that case, makes sense to move raw_spin_lock in
> sugov_update_shared further down? (Just discussing, this point is
> independent of your patch), Something like:

Even that was discussed tomorrow with Peter :)

No it wouldn't work because sg_cpu->util we are updating here may be
getting read from some other cpu that shares policy with sg_cpu.

-- 
viresh

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Viresh Kumar]

On 26-07-17, 23:13, Joel Fernandes (Google) wrote:
> On Wed, Jul 26, 2017 at 10:50 PM, Viresh Kumar  
> wrote:
> > On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
> >> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  
> >> wrote:
> >> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct 
> >> > update_util_data *hook, u64 time,
> >> > sugov_set_iowait_boost(sg_cpu, time, flags);
> >> > sg_cpu->last_update = time;
> >> >
> >> > -   if (!sugov_should_update_freq(sg_policy, time))
> >> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
> >> > return;
> >>
> >> Since with the remote callbacks now possible, isn't it unsafe to
> >> modify sg_cpu and sg_policy structures without a lock in
> >> sugov_update_single?
> >>
> >> Unlike sugov_update_shared, we don't acquire any lock in
> >> sugov_update_single before updating these structures. Did I miss
> >> something?
> >
> > As Peter already mentioned it earlier, the callbacks are called with
> > rq locks held and so sugov_update_single() wouldn't get called in
> > parallel for a target CPU.
> 
> Ah ok, I have to catch up with that discussion since I missed the
> whole thing. Now that you will have me on CC, that shouldn't happen,
> thanks and sorry about the noise.
> 
> > That's the only race you were worried about ?
> 
> Yes. So then in that case, makes sense to move raw_spin_lock in
> sugov_update_shared further down? (Just discussing, this point is
> independent of your patch), Something like:

Even that was discussed tomorrow with Peter :)

No it wouldn't work because sg_cpu->util we are updating here may be
getting read from some other cpu that shares policy with sg_cpu.

-- 
viresh

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Joel Fernandes (Google)]

On Wed, Jul 26, 2017 at 10:50 PM, Viresh Kumar  wrote:
> On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
>> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  
>> wrote:
>> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct 
>> > update_util_data *hook, u64 time,
>> > sugov_set_iowait_boost(sg_cpu, time, flags);
>> > sg_cpu->last_update = time;
>> >
>> > -   if (!sugov_should_update_freq(sg_policy, time))
>> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
>> > return;
>>
>> Since with the remote callbacks now possible, isn't it unsafe to
>> modify sg_cpu and sg_policy structures without a lock in
>> sugov_update_single?
>>
>> Unlike sugov_update_shared, we don't acquire any lock in
>> sugov_update_single before updating these structures. Did I miss
>> something?
>
> As Peter already mentioned it earlier, the callbacks are called with
> rq locks held and so sugov_update_single() wouldn't get called in
> parallel for a target CPU.

Ah ok, I have to catch up with that discussion since I missed the
whole thing. Now that you will have me on CC, that shouldn't happen,
thanks and sorry about the noise.

> That's the only race you were worried about ?

Yes. So then in that case, makes sense to move raw_spin_lock in
sugov_update_shared further down? (Just discussing, this point is
independent of your patch), Something like:

diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c
index 622eed1b7658..9a6c12fb2c16 100644
--- a/kernel/sched/cpufreq_schedutil.c
+++ b/kernel/sched/cpufreq_schedutil.c
@@ -295,8 +295,6 @@ static void sugov_update_shared(struct
update_util_data *hook, u64 time,

sugov_get_util(, );

-   raw_spin_lock(_policy->update_lock);
-
sg_cpu->util = util;
sg_cpu->max = max;
sg_cpu->flags = flags;
@@ -304,6 +302,8 @@ static void sugov_update_shared(struct
update_util_data *hook, u64 time,
sugov_set_iowait_boost(sg_cpu, time, flags);
sg_cpu->last_update = time;

+   raw_spin_lock(_policy->update_lock);
+
if (sugov_should_update_freq(sg_policy, time)) {
if (flags & SCHED_CPUFREQ_RT_DL)
next_f = sg_policy->policy->cpuinfo.max_freq;



thanks,

-Joel

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Joel Fernandes (Google)]

On Wed, Jul 26, 2017 at 10:50 PM, Viresh Kumar  wrote:
> On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
>> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  
>> wrote:
>> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct 
>> > update_util_data *hook, u64 time,
>> > sugov_set_iowait_boost(sg_cpu, time, flags);
>> > sg_cpu->last_update = time;
>> >
>> > -   if (!sugov_should_update_freq(sg_policy, time))
>> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
>> > return;
>>
>> Since with the remote callbacks now possible, isn't it unsafe to
>> modify sg_cpu and sg_policy structures without a lock in
>> sugov_update_single?
>>
>> Unlike sugov_update_shared, we don't acquire any lock in
>> sugov_update_single before updating these structures. Did I miss
>> something?
>
> As Peter already mentioned it earlier, the callbacks are called with
> rq locks held and so sugov_update_single() wouldn't get called in
> parallel for a target CPU.

Ah ok, I have to catch up with that discussion since I missed the
whole thing. Now that you will have me on CC, that shouldn't happen,
thanks and sorry about the noise.

> That's the only race you were worried about ?

Yes. So then in that case, makes sense to move raw_spin_lock in
sugov_update_shared further down? (Just discussing, this point is
independent of your patch), Something like:

diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c
index 622eed1b7658..9a6c12fb2c16 100644
--- a/kernel/sched/cpufreq_schedutil.c
+++ b/kernel/sched/cpufreq_schedutil.c
@@ -295,8 +295,6 @@ static void sugov_update_shared(struct
update_util_data *hook, u64 time,

sugov_get_util(, );

-   raw_spin_lock(_policy->update_lock);
-
sg_cpu->util = util;
sg_cpu->max = max;
sg_cpu->flags = flags;
@@ -304,6 +302,8 @@ static void sugov_update_shared(struct
update_util_data *hook, u64 time,
sugov_set_iowait_boost(sg_cpu, time, flags);
sg_cpu->last_update = time;

+   raw_spin_lock(_policy->update_lock);
+
if (sugov_should_update_freq(sg_policy, time)) {
if (flags & SCHED_CPUFREQ_RT_DL)
next_f = sg_policy->policy->cpuinfo.max_freq;



thanks,

-Joel

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Viresh Kumar]

On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  wrote:
> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct update_util_data 
> > *hook, u64 time,
> > sugov_set_iowait_boost(sg_cpu, time, flags);
> > sg_cpu->last_update = time;
> >
> > -   if (!sugov_should_update_freq(sg_policy, time))
> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
> > return;
> 
> Since with the remote callbacks now possible, isn't it unsafe to
> modify sg_cpu and sg_policy structures without a lock in
> sugov_update_single?
> 
> Unlike sugov_update_shared, we don't acquire any lock in
> sugov_update_single before updating these structures. Did I miss
> something?

As Peter already mentioned it earlier, the callbacks are called with
rq locks held and so sugov_update_single() wouldn't get called in
parallel for a target CPU.

That's the only race you were worried about ?

-- 
viresh

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Viresh Kumar]

On 26-07-17, 22:34, Joel Fernandes (Google) wrote:
> On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  wrote:
> > @@ -221,7 +226,7 @@ static void sugov_update_single(struct update_util_data 
> > *hook, u64 time,
> > sugov_set_iowait_boost(sg_cpu, time, flags);
> > sg_cpu->last_update = time;
> >
> > -   if (!sugov_should_update_freq(sg_policy, time))
> > +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
> > return;
> 
> Since with the remote callbacks now possible, isn't it unsafe to
> modify sg_cpu and sg_policy structures without a lock in
> sugov_update_single?
> 
> Unlike sugov_update_shared, we don't acquire any lock in
> sugov_update_single before updating these structures. Did I miss
> something?

As Peter already mentioned it earlier, the callbacks are called with
rq locks held and so sugov_update_single() wouldn't get called in
parallel for a target CPU.

That's the only race you were worried about ?

-- 
viresh

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Joel Fernandes (Google)]

Hi Viresh,

On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  wrote:
> We do not call cpufreq callbacks from scheduler core for remote
> (non-local) CPUs currently. But there are cases where such remote
> callbacks are useful, specially in the case of shared cpufreq policies.
>
> This patch updates the scheduler core to call the cpufreq callbacks for
> remote CPUs as well.
>
> For now, all the registered utilization update callbacks are updated to
> return early if remote callback is detected. That is, this patch just
> moves the decision making down in the hierarchy.
>
> Later patches would enable remote callbacks for shared policies.
>
> Based on initial work from Steve Muckle.
>
> Signed-off-by: Viresh Kumar 

> --- a/kernel/sched/cpufreq_schedutil.c
> +++ b/kernel/sched/cpufreq_schedutil.c
> @@ -72,10 +72,15 @@ static DEFINE_PER_CPU(struct sugov_cpu, sugov_cpu);
>
>  / Governor internals ***/
>
> -static bool sugov_should_update_freq(struct sugov_policy *sg_policy, u64 
> time)
> +static bool sugov_should_update_freq(struct sugov_policy *sg_policy, u64 
> time,
> +int target_cpu)
>  {
> s64 delta_ns;
>
> +   /* Don't allow remote callbacks */
> +   if (smp_processor_id() != target_cpu)
> +   return false;
> +
> if (sg_policy->work_in_progress)
> return false;
>
> @@ -221,7 +226,7 @@ static void sugov_update_single(struct update_util_data 
> *hook, u64 time,
> sugov_set_iowait_boost(sg_cpu, time, flags);
> sg_cpu->last_update = time;
>
> -   if (!sugov_should_update_freq(sg_policy, time))
> +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
> return;

Since with the remote callbacks now possible, isn't it unsafe to
modify sg_cpu and sg_policy structures without a lock in
sugov_update_single?

Unlike sugov_update_shared, we don't acquire any lock in
sugov_update_single before updating these structures. Did I miss
something?


thanks,

-Joel

Re: [Eas-dev] [PATCH V4 1/3] sched: cpufreq: Allow remote cpufreq callbacks

[Joel Fernandes (Google)]

Hi Viresh,

On Wed, Jul 26, 2017 at 2:22 AM, Viresh Kumar  wrote:
> We do not call cpufreq callbacks from scheduler core for remote
> (non-local) CPUs currently. But there are cases where such remote
> callbacks are useful, specially in the case of shared cpufreq policies.
>
> This patch updates the scheduler core to call the cpufreq callbacks for
> remote CPUs as well.
>
> For now, all the registered utilization update callbacks are updated to
> return early if remote callback is detected. That is, this patch just
> moves the decision making down in the hierarchy.
>
> Later patches would enable remote callbacks for shared policies.
>
> Based on initial work from Steve Muckle.
>
> Signed-off-by: Viresh Kumar 

> --- a/kernel/sched/cpufreq_schedutil.c
> +++ b/kernel/sched/cpufreq_schedutil.c
> @@ -72,10 +72,15 @@ static DEFINE_PER_CPU(struct sugov_cpu, sugov_cpu);
>
>  / Governor internals ***/
>
> -static bool sugov_should_update_freq(struct sugov_policy *sg_policy, u64 
> time)
> +static bool sugov_should_update_freq(struct sugov_policy *sg_policy, u64 
> time,
> +int target_cpu)
>  {
> s64 delta_ns;
>
> +   /* Don't allow remote callbacks */
> +   if (smp_processor_id() != target_cpu)
> +   return false;
> +
> if (sg_policy->work_in_progress)
> return false;
>
> @@ -221,7 +226,7 @@ static void sugov_update_single(struct update_util_data 
> *hook, u64 time,
> sugov_set_iowait_boost(sg_cpu, time, flags);
> sg_cpu->last_update = time;
>
> -   if (!sugov_should_update_freq(sg_policy, time))
> +   if (!sugov_should_update_freq(sg_policy, time, hook->cpu))
> return;

Since with the remote callbacks now possible, isn't it unsafe to
modify sg_cpu and sg_policy structures without a lock in
sugov_update_single?

Unlike sugov_update_shared, we don't acquire any lock in
sugov_update_single before updating these structures. Did I miss
something?


thanks,

-Joel