On 10/08/2011 03:25 AM, Michael Ost wrote: > Hi list, > > We are seeing unexpected interruptions of SCHED_RR audio processing > threads, and are struggling to understand why they are happening. Does > anyone have any good tips or tools to suggest to help figure out what is > preempting or delaying realtime audio threads?
https://www.osadl.org/Realtime-Preempt-Kernel.kernel-rt.0.html#builtintools It's a bit dated but so is your 2.6.24.3 kernel. You will need to compile the kernel with CONFIG_PREEMPT_TRACER=y CONFIG_SCHED_TRACER=y under "Kernel hacking -> Tracers -> .." to get access to /sys/kernel/debug/tracing/ The man page of cyclictest (8) includes some hints: see the '-b' option on how to produce traces from the scheduler. Also check out ..linux-rt-source/Documentation/trace/histograms.txt > The issues are coming up with Receptor [see (*) below for an intro] > running its 2.6.24.3 CCRMA based kernel. The bug appears with Receptor > in its "dual core mode", with three instances of Native Instruments' > Kontakt 4 in DFD + multi-core mode. Either lots of held notes, or large > patches are needed to get the spikes. > > With these settings there are 5 SCHED_RR threads processing audio (on a > two-core system). 2 are from Receptor, and 1 from each instance of > Kontakt. These Kontakt "helper threads" are released to do work as > possible while the audio thread is processing. > > Kontakt/DFD is using mmap to bring its files into memory. This is done > in a lower priority "DFD thread", and the mapped memory is used by the > r/t audio threads. > > DFD is important because the problems don't happen without it. And the > high SCHED_RR thread count is important, because the problems also don't > happen if we reduce the count. is the DFD thread calling mlock() on the mapped memory? maybe madvise/mlock fail because you're trying to lock more pages than RLIMIT_MEMLOCK permits? just a guess. Documentation/trace/mmiotrace.txt should help to find out if a process blocks due to memory mapped i/o. The debug interface is nifty and the time consuming part is compiling a kernel with CONFIG_MMIOTRACE=y :) > When the spike happens there are no: > * wine bottlenecks > * system calls > * threads blocking on each other > * page faults during audio processing > > There do appear to be "involuntary context switches" (as reported by > getrusage) when the spikes happen. This makes it seem like the scheduler > is interrupting our threads. But how do you figure out why that is > happening? > > There aren't many threads in the system with higher priority. All of the > 5 processing threads are SCHED_RR/76. The higher priority threads in the > system are: > * migration/0 - FIFO/99 > * migration/1 - FIFO/99 > * watchdog/0 - FIFO/99 > * watchdog/1 - FIFO/99 > * posix_cpu_timer (x2) - FIFO/99 > * IRQ8 (rtc) - FIFO/99 > * IRQ20 (our audio card) - FIFO 77 > > Could other kernel activity interrupt the audio threads? Are there > issues with memory mapping, that can block other unrelated threads? Are > there just too danged many SCHED_RR threads fighting for two cores? > > Anyone have any suggestions for how to trace the scheduler, and thread > or process interruptions? > > Apologies for the lengthy post, but this is a tricky subject. > > Thanks for any tips or insights, It would not surprise me if that is one of the many issues that got fixed since 2.6.24. That kernel still featured the BKL [BigKernelLock] and the problem you are describing is not too far out.. 3.0.6-rt17 requires a bit more work, but 2.6.39 is very stable. > Michael Ost > Muse Research & Development > > (*) Receptor (http://www.museresearch.com/products/receptor.php) is a > standalone hardware VST plugin player. It runs Windows VSTs using Linux > and Wine in a proprietary host program. We have patched Wine > (ftp://anonymous+museresearch....@ftp.museresearch.com/rpms/1.8/wine-1.1.7-muse16.fc8.src.rpm) > to make it work better with VSTs in our environment. > > _______________________________________________ > Linux-audio-dev mailing list > Linux-audio-dev@lists.linuxaudio.org > http://lists.linuxaudio.org/listinfo/linux-audio-dev _______________________________________________ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org http://lists.linuxaudio.org/listinfo/linux-audio-dev