[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-11-13 Thread Steve Langasek 
** Changed in: sysvinit (Ubuntu Trusty)
Milestone: None => ubuntu-14.04.4

** Summary changed:

- On demand cpufreq govneror causes large amounts of jitter
+ On demand cpufreq governor causes large amounts of jitter

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq governor causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Fix Released
Status in sysvinit source package in Trusty:
  Triaged
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-11-02 Thread bugproxy 
** Tags removed: targetmilestone-inin1404

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Fix Released
Status in sysvinit source package in Trusty:
  Triaged
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or work-queue only to change PState and not really for
  estimation of load.  Hence steady state load will experience zero

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-11-02 Thread Michael Hohnbaum 
** Tags added: targetmilestone-inin1404

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Fix Released
Status in sysvinit source package in Trusty:
  Triaged
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or work-queue only to change PState and not really for
  estimation of load.  Hence steady state load will experience zero
  jitter

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-21 Thread Steve Langasek 
** Changed in: sysvinit (Ubuntu Trusty)
 Assignee: (unassigned) => Adam Conrad (adconrad)

** Changed in: sysvinit (Ubuntu Trusty)
   Status: New => Triaged

** Changed in: sysvinit (Ubuntu Trusty)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Fix Released
Status in sysvinit source package in Trusty:
  Triaged
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-19 Thread Breno Leitão 
I just tested it on Wily and it is setting the sampling_down_factor
properly. Thanks!

$ cat /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
100
$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Fix Released
Status in sysvinit source package in Trusty:
  New
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-16 Thread Launchpad Bug Tracker 
** Branch linked: lp:ubuntu/wily-proposed/sysvinit

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Triaged
Status in sysvinit source package in Trusty:
  New
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or work-queue only to change PState and not really for
  estimation of load.  Hence steady state load will experience zero

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-16 Thread Launchpad Bug Tracker 
This bug was fixed in the package sysvinit - 2.88dsf-59.2ubuntu2

---
sysvinit (2.88dsf-59.2ubuntu2) wily; urgency=medium

  * Adjust sampling_down_factor to 100 on ppc64 kernels (LP: #1483586)

 -- Adam Conrad   Thu, 15 Oct 2015 20:43:12 -0600

** Changed in: sysvinit (Ubuntu)
   Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Fix Released
Status in sysvinit source package in Trusty:
  New
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-15 Thread Adam Conrad 
** Also affects: sysvinit (Ubuntu Vivid)
   Importance: Undecided
   Status: New

** Also affects: sysvinit (Ubuntu Trusty)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Triaged
Status in sysvinit source package in Trusty:
  New
Status in sysvinit source package in Vivid:
  New

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-07 Thread Adam Conrad 
Sure, we can get it fixed for 15.10 (and SRUed back).

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Triaged

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or work-queue only to change PState and not really for
  estimation of load.  Hence steady state load will experience zero
  jitter from cpufreq.

  --Vaidy

  
  == Comment: #7 - Shilpasri G. Bhat  -

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-10-05 Thread Breno Leitão 
Hi Adam,

I understand that we will not be able to have this fixed by the 15.10
release, right?

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Triaged

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or work-queue only to change PState and not really for
  estimation of load.  Hence steady state load will experience zero
  jitter from cpufreq.

  --Vaidy

  
  == Comment: #7 -

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-09-28 Thread bugproxy 
** Tags removed: targetmilestone-inin1504
** Tags added: targetmilestone-inin1510

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Triaged

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but still be able to set per-core PState. We need to take
  an interrupt or work-queue only to change PState and not really for
  estimation of load.  Hence steady state load will experience zero
  jitter from cpufreq.

  --Vaidy

  
  == Comment: #7 - Shilpasri G. Bhat

[Touch-packages] [Bug 1483586] Re: On demand cpufreq govneror causes large amounts of jitter

2015-09-09 Thread Steve Langasek 
Assigning to Foundations for the init script workaround.

** Package changed: ubuntu => sysvinit (Ubuntu)

** Changed in: sysvinit (Ubuntu)
   Importance: Undecided => High

** Changed in: sysvinit (Ubuntu)
   Status: New => Triaged

** Changed in: sysvinit (Ubuntu)
 Assignee: Taco Screen team (taco-screen-team) => Adam Conrad (adconrad)

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to sysvinit in Ubuntu.
https://bugs.launchpad.net/bugs/1483586

Title:
  On demand cpufreq govneror causes large amounts of jitter

Status in sysvinit package in Ubuntu:
  Triaged

Bug description:
  == Comment: #0 - Anton Blanchard  - 2015-07-16 22:22:09 ==
  We are seeing large amounts of jitter caused by od_dbs_timer(). We should 
slow down the rate of updates and or turn this into a timer. Having a workqueue 
execute so often is very noticeable.

  # echo 1 >
  /sys/kernel/debug/tracing/events/workqueue/workqueue_execute_start/enable

  (wait a while)

  # cat /sys/kernel/debug/trace

 <...>-67605 [040]  849622.393576: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.403574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.403575: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.413574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.413575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.423575: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.433574: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer
 <...>-67605 [040]  849622.433574: workqueue_execute_start: 
work struct c007fba1ba20: function od_dbs_timer
 <...>-116685 [048]  849622.443573: workqueue_execute_start: 
work struct c007fbc1ba20: function od_dbs_timer

  == Comment: #1 - Shilpasri G. Bhat  - 2015-07-22 
19:42:38 ==
  Hi Anton,

  We can set the governor's tunable 'sampling_down_factor' to decrease
  the rate of updates. When this tunable is set to a value greater than
  1, the sampling period of the governor is increased during the peak
  load to sampling_period times sampling_down_factor. This will reduce
  the jitter caused by od_dbs_timer() when the cpu is busy.

  I am currently running benchmarks to find out the optimal value for
  this tunable and will post them soon.

  Thanks and Regards,
  Shilpa

  == Comment: #2 - Anton Blanchard  - 2015-07-31 03:44:49 ==
  FYI We are also seeing high levels of CPU consumed by this on a LAMP workload:

   2.54%  kworker/0:0  [kernel.kallsyms] [k] osq_lock
  |   
  ---osq_lock
 |   
 |--99.83%-- mutex_optimistic_spin
 |  __mutex_lock_slowpath
 |  mutex_lock
 |  |  
 |  |--80.08%-- od_dbs_timer

  2.5% of total CPU time spent in the od_dbs_timer mutex.

  == Comment: #3 - Anton Blanchard  - 2015-07-31 06:00:45 ==
  Hitting this on a customer setup, raising priority

  == Comment: #4 - Shilpasri G. Bhat  - 2015-08-03 
06:47:40 ==
  I used `perf top` and `perf record` to observe the overhead caused by 
'osq_lock'.
  Both with ebizzy and SPECPower's ssjb workload I am able to see an overhead 
of 0.03% caused by 'osq_lock' with default governor settings. 
  With sampling_down_factor=100, (1second) I am able to see 0.00% of overhead 
by 'osq_lock'.

  So this might not be a good data point to showcase, but by reducing
  the  od_dbs_timer interrupts we are guaranteed to decrease the
  overhead caused by 'osq_lock'.

  == Comment: #5 - VAIDYANATHAN SRINIVASAN  - 2015-08-03 
09:09:09 ==
  Hi Anton,

  Thanks for opening the bz to track and fix this issue.  Shilpa is
  trying different workarounds.  Here is our plan:

  (1) Use sampling_down_factor and other tunables in current Ubuntu
  releases to workaround the issue or minimise the impact.

  (2) Redesign cpufreq subsystem on powerpc similar to intel pstate
  driver so that we can program timers and cancel them dynamically based
  on different utilization points.  Target Ubuntu 16.04 and then
  backport to 14.04.x and other distros.

  (3) Enhance design for (2) buy estimating core level utilization
  without running timers in each thread and then decide the target
  PState

  (4) Explore hardware assist so that we can avoid per-core estimation
  in software but