Re: ULE realtime scheduler advice needed
On 11/22/2022 5:00 PM, Tomoaki AOKI wrote:
A long shot, but I am curious if by chance you have hwpstate_intel for
your cpu frequency driver. If so, does setting
dev.hwpstate_intel.0.epp=0 make any difference ?
Yes, I have four of those, set to 50 by default. Let me try.
--HPS
FYI: I habitally run below manually (as root) when I'm on AC powerline.
sysctl -aN | fgrep dev.hwpstate | fgrep epp | while read OID ; do ; \
sysctl ${OID}=0 ; done
It was a bit of a POLA the first time I encountered this driver going
from 12 to RELENG_13. Same hardware behaved rather differently as its a
different power profile default than what I expected and for my use case
(firewall and router) it was causing dropped packets
---Mike
Re: ULE realtime scheduler advice needed
On Tue, 22 Nov 2022 22:38:04 +0100
Hans Petter Selasky wrote:
> On 11/22/22 20:28, mike tancsa wrote:
> > On 11/17/2022 11:47 PM, Hans Petter Selasky wrote:
> >> Hi,
> >>
> >> I'm doing some work with audio and have noticed some problems with the
> >> ULE scheduler. I have a program that generate audio based on
> >> key-presses. When no keys are pressed, the load is near 0%, but as
> >> soon as you start pressing keys, the load goes maybe to 80% of a CPU
> >> core. This program I run with rtprio 8 xxx. The issue I observe or
> >> hear actually, is that it takes too long until the scheduler grasps
> >> that this program needs it's own CPU core and stops time-sharing the
> >> program. When I however use cpuset -l xxx rtprio 8 yyy everything is
> >> good, and the program outputs realtime audio in-time.
> >>
> >> Or is this perhaps a CPU frequency stepping issue?
> >>
> >> Any advice on where to look?
> >>
> > A long shot, but I am curious if by chance you have hwpstate_intel for
> > your cpu frequency driver. If so, does setting
> > dev.hwpstate_intel.0.epp=0 make any difference ?
> >
>
> Yes, I have four of those, set to 50 by default. Let me try.
>
> --HPS
FYI: I habitally run below manually (as root) when I'm on AC powerline.
sysctl -aN | fgrep dev.hwpstate | fgrep epp | while read OID ; do ; \
sysctl ${OID}=0 ; done
--
Tomoaki AOKI
Re: ULE realtime scheduler advice needed
On 11/22/22 20:28, mike tancsa wrote: On 11/17/2022 11:47 PM, Hans Petter Selasky wrote: Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. Or is this perhaps a CPU frequency stepping issue? Any advice on where to look? A long shot, but I am curious if by chance you have hwpstate_intel for your cpu frequency driver. If so, does setting dev.hwpstate_intel.0.epp=0 make any difference ? Yes, I have four of those, set to 50 by default. Let me try. --HPS
Re: ULE realtime scheduler advice needed
On 11/17/2022 11:47 PM, Hans Petter Selasky wrote: Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. Or is this perhaps a CPU frequency stepping issue? Any advice on where to look? A long shot, but I am curious if by chance you have hwpstate_intel for your cpu frequency driver. If so, does setting dev.hwpstate_intel.0.epp=0 make any difference ? ---Mike
Re: ULE realtime scheduler advice needed
On Tue, 22 Nov 2022 13:15:33 +0100 Johan Hendriks wrote: > > On 21/11/2022 21:24, Hans Petter Selasky wrote: > > On 11/21/22 20:12, Mark Johnston wrote: > >> There were some bug fixes earlier this year to address problems where > >> high-priority threads were not getting scheduled quickly enough. If > >> you're using an old kernel, they might improve things: > > > > Are any of these fixes merged to stable/13 ? > > > > --HPS > > > It looks like it. > https://freshbsd.org/freebsd/src?q=sched_ule.c Or track PR on Bugzilla via commit logs on main (pointed by markj@) instead of Differential Revision on Phablicator. Phab shouldn't know about branches other than main this case. On Bugzilla, we can see each of them MFC'ed to stable/13 with second commit-hook. -- Tomoaki AOKI
Re: ULE realtime scheduler advice needed
On 21/11/2022 21:24, Hans Petter Selasky wrote: On 11/21/22 20:12, Mark Johnston wrote: There were some bug fixes earlier this year to address problems where high-priority threads were not getting scheduled quickly enough. If you're using an old kernel, they might improve things: Are any of these fixes merged to stable/13 ? --HPS It looks like it. https://freshbsd.org/freebsd/src?q=sched_ule.c
Re: ULE realtime scheduler advice needed
On 11/21/22 20:12, Mark Johnston wrote: There were some bug fixes earlier this year to address problems where high-priority threads were not getting scheduled quickly enough. If you're using an old kernel, they might improve things: Are any of these fixes merged to stable/13 ? --HPS
Re: ULE realtime scheduler advice needed
On Fri, Nov 18, 2022 at 05:47:58AM +0100, Hans Petter Selasky wrote: > Hi, > > I'm doing some work with audio and have noticed some problems with the > ULE scheduler. I have a program that generate audio based on > key-presses. When no keys are pressed, the load is near 0%, but as soon > as you start pressing keys, the load goes maybe to 80% of a CPU core. > This program I run with rtprio 8 xxx. The issue I observe or hear > actually, is that it takes too long until the scheduler grasps that this > program needs it's own CPU core and stops time-sharing the program. When > I however use cpuset -l xxx rtprio 8 yyy everything is good, and the > program outputs realtime audio in-time. > > Or is this perhaps a CPU frequency stepping issue? > > Any advice on where to look? There were some bug fixes earlier this year to address problems where high-priority threads were not getting scheduled quickly enough. If you're using an old kernel, they might improve things: https://cgit.freebsd.org/src/commit/?id=0927ff78147b4d00a75054bbad299946208e1e91 https://cgit.freebsd.org/src/commit/?id=6d3f74a14a83b867c273c6be2599da182a9b9ec7 https://cgit.freebsd.org/src/commit/?id=03f868b163ad46d6f7cb03dc46fb83ca01fb8f69 https://cgit.freebsd.org/src/commit/?id=a889a65ba36985dfb3ac1607be35ca2b2c8c Otherwise, you might try using tools/sched/schedgraph.d and schedgraph.py to visualize what's going on. schedgraph.d logs all scheduler events of interest and schedgraph.py presents a row for each CPU and thread in the system, making it easier to see why threads are waiting on runqueues.
Re: ULE realtime scheduler advice needed
I've lost track with, but IIRC, someone wrote here, or other ML, or even forums, kern.sched.preempt_thresh=224 was the default for PC-BSD. And found some discussion started at [1] on freebsd-stable ML in Apr. 2018. One more place is at forums [2]. Sorry, not read all of them to confirm. [1] https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088678.html [2] https://forums.freebsd.org/threads/general-freebsd-desktop-workload-optimization-thread.21853/ On Sat, 19 Nov 2022 10:28:50 +0100 Hans Petter Selasky wrote: > Hi Alexander, > > Thank you for the pointers. > > I will try it out. > > --HPS > > On 11/18/22 09:18, Alexander Leidinger wrote: > > Quoting Hans Petter Selasky (from Fri, 18 Nov 2022 > > 05:47:58 +0100): > > > >> Hi, > >> > >> I'm doing some work with audio and have noticed some problems with the > >> ULE scheduler. I have a program that generate audio based on > >> key-presses. When no keys are pressed, the load is near 0%, but as > >> soon as you start pressing keys, the load goes maybe to 80% of a CPU > >> core. This program I run with rtprio 8 xxx. The issue I observe or > >> hear actually, is that it takes too long until the scheduler grasps > >> that this program needs it's own CPU core and stops time-sharing the > >> program. When I however use cpuset -l xxx rtprio 8 yyy everything is > >> good, and the program outputs realtime audio in-time. > > > > I have something in my mind about ULE not handling idleprio and/or > > rtprio correctly, but I have no pointer to a validation of this. > > > >> Or is this perhaps a CPU frequency stepping issue? > > > > You could play with > > rc.conf (/etc/rc.d/power_profile): > > performance_cpu_freq="HIGH" > > performance_cx_lowest="C3" # see sysctl hw.cpu.0 | grep cx > > economy_cx_lowest="C3" # see sysctl hw.cpu.0 | grep cx > > > > Your system may provide other Cx possibilities, and ging to a lower > > number (e.g. C1) means less power-saving but faster response from the > > CPU (I do not expect that this is causing the issue you have). > > > >> Any advice on where to look? > > > > Potential sysctl to play with to change "interactivity detection" in ULE: > > https://www.mail-archive.com/freebsd-stable@freebsd.org/msg112118.html > > > > Bye, > > Alexander. > > > -- Tomoaki AOKI
Re: ULE realtime scheduler advice needed
Hi Alexander, Thank you for the pointers. I will try it out. --HPS On 11/18/22 09:18, Alexander Leidinger wrote: Quoting Hans Petter Selasky (from Fri, 18 Nov 2022 05:47:58 +0100): Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. I have something in my mind about ULE not handling idleprio and/or rtprio correctly, but I have no pointer to a validation of this. Or is this perhaps a CPU frequency stepping issue? You could play with rc.conf (/etc/rc.d/power_profile): performance_cpu_freq="HIGH" performance_cx_lowest="C3" # see sysctl hw.cpu.0 | grep cx economy_cx_lowest="C3" # see sysctl hw.cpu.0 | grep cx Your system may provide other Cx possibilities, and ging to a lower number (e.g. C1) means less power-saving but faster response from the CPU (I do not expect that this is causing the issue you have). Any advice on where to look? Potential sysctl to play with to change "interactivity detection" in ULE: https://www.mail-archive.com/freebsd-stable@freebsd.org/msg112118.html Bye, Alexander.
Re: ULE realtime scheduler advice needed
Quoting Hans Petter Selasky (from Fri, 18 Nov 2022 05:47:58 +0100): Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. I have something in my mind about ULE not handling idleprio and/or rtprio correctly, but I have no pointer to a validation of this. Or is this perhaps a CPU frequency stepping issue? You could play with rc.conf (/etc/rc.d/power_profile): performance_cpu_freq="HIGH" performance_cx_lowest="C3" # see sysctl hw.cpu.0 | grep cx economy_cx_lowest="C3" # see sysctl hw.cpu.0 | grep cx Your system may provide other Cx possibilities, and ging to a lower number (e.g. C1) means less power-saving but faster response from the CPU (I do not expect that this is causing the issue you have). Any advice on where to look? Potential sysctl to play with to change "interactivity detection" in ULE: https://www.mail-archive.com/freebsd-stable@freebsd.org/msg112118.html Bye, Alexander. -- http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.orgnetch...@freebsd.org : PGP 0x8F31830F9F2772BF pgpkBXOWiYjFT.pgp Description: Digitale PGP-Signatur
ULE realtime scheduler advice needed
Hi, I'm doing some work with audio and have noticed some problems with the ULE scheduler. I have a program that generate audio based on key-presses. When no keys are pressed, the load is near 0%, but as soon as you start pressing keys, the load goes maybe to 80% of a CPU core. This program I run with rtprio 8 xxx. The issue I observe or hear actually, is that it takes too long until the scheduler grasps that this program needs it's own CPU core and stops time-sharing the program. When I however use cpuset -l xxx rtprio 8 yyy everything is good, and the program outputs realtime audio in-time. Or is this perhaps a CPU frequency stepping issue? Any advice on where to look? --HPS
