Re: [linux-audio-dev] kernel preemption patch and 2.4.19
On Wednesday 02 June 2004 22.13, Michael Ost wrote: What specific benefits have folks seen by turning on the kernel preemption patch in a 2.4.19 kernel? 2.4 series is not really built for preemption. But... We found that a nasty system crash was fixed by turning off preemption. The crash would happen fairly reliably by switching between virtual terminals a number of times. It locked up the system hard. So hard that we can't really find the problem in the kernel; we just found a work-around basically by trial and error. Do you use a SMP computer? If not then you might have run into something that could hurt a SMP computer running without preemption... (Running SMP and preemption is kind of weird. And you may run into additional problems - per CPU data has also to be protected by spinlocks, etc... ) We turned on kernel preemption basically because everyone was doing it. We haven't noticed any obvious, serious problems or differences since turning preemption off. Does anyone have some suggestions about what differences to expect and what we might have a closer look at? You won't notice much difference unless you run real time processes. But they will need to be suid root (or some wrapper/helper) to become RT processes, and many kernels do not ship with multimedia stuff suid... Also, we are newbies about reporting this kind of crash. Any clues about where to report it or ask about it? First - you mention virtual terminals What video driver are you using? Nvidia? If that is the case retest with xfree86 (nv) Second - try to what is happening (Magic SysRq) Third - 2.4.19 is kind of old now (not that many will be interested...), upgrade and try to reproduce. /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Latencytest of filesystems (ext2, ext3 w. defaults)
Result looks OK with a SuSE kernel (I think they have lowlatency patch
applied), worst was
ext2 - it was diskread that had most problems...
ext3 - diskread (!?)
diskrm included for reference
http://w1.910.telia.com/~u91005836/index.html
[I have to recheck the results when I am awake :-]
Patches are attached - with extras (see below).
Hmm... the output should really add
kernel version: uname -r
2.4.21-215-athlon
Close, but does not tell that it is precompiled by SuSE
X11 server and filesystem type: how to find that automatically?
I use, XFree86 nv
Extras:
* some filehandling (non local root of testing) and
* run with min(SCHED_FIFO)+1 not max
- it is a CPU hog (80%) [the more CPU usage the less prio]
- show that it works to avoid creating a trend...
/RogerL
--
Roger Larsson
Skellefteå
Sweden
diff -Naur latencytest0.42-png/cpu_latency.c latencytest0.42-png+rl/cpu_latency.c
--- latencytest0.42-png/cpu_latency.c 1999-06-27 15:44:20.0 +0200
+++ latencytest0.42-png+rl/cpu_latency.c 2004-06-03 01:38:31.0 +0200
@@ -217,7 +217,7 @@
* set the process to realtime privs
*/
memset(schp, 0, sizeof(schp));
- schp.sched_priority = sched_get_priority_max(SCHED_FIFO);
+ schp.sched_priority = sched_get_priority_min(SCHED_FIFO)+1;
if (sched_setscheduler(0, SCHED_FIFO, schp) != 0) {
perror(sched_setscheduler);
diff -Naur latencytest0.42-png/diskstress latencytest0.42-png+rl/diskstress
--- latencytest0.42-png/diskstress 1999-06-04 01:29:42.0 +0200
+++ latencytest0.42-png+rl/diskstress 2004-06-03 01:23:18.0 +0200
@@ -1,6 +1,8 @@
+export latdiskroot=. # or path to other filesystem type (writeable)
+
echo creating tmpfile
-head -c 6000 /dev/zero tmpfile
-echo tmpfile creation done
+head -c 6000 /dev/zero $latdiskroot/tmpfile
+echo $latdiskroot/tmpfile creation done
echo starting 4 find processes
find /usr /dev/null 2/dev/null
sleep 2
@@ -11,10 +13,13 @@
find /home /dev/null 2/dev/null
sleep 2
echo find processes started
-echo copying tmpfile to tmpfile2
-cp tmpfile tmpfile2
-echo copying tmpfile and tmpfile2 to /dev/null
-cat tmpfile tmpfile2 /dev/null
+echo copying $latdiskroot/tmpfile to $latdiskroot/tmpfile2
+cp $latdiskroot/tmpfile $latdiskroot/tmpfile2
+echo copying $latdiskroot/tmpfile and $latdiskroot/tmpfile2 to /dev/null
+cat $latdiskroot/tmpfile $latdiskroot/tmpfile2 /dev/null
echo copy done
+echo remove $latdiskroot/tmpfile and $latdiskroot/tmpfile2
+rm $latdiskroot/tmpfile $latdiskroot/tmpfile2
+echo remove done
killall find
diff -Naur latencytest0.42-png/do_tests latencytest0.42-png+rl/do_tests
--- latencytest0.42-png/do_tests 2000-07-25 11:37:58.0 +0200
+++ latencytest0.42-png+rl/do_tests 2004-06-03 01:37:12.0 +0200
@@ -1,3 +1,6 @@
+export latdiskroot=. # or path to other filesystem type (writeable)
+
+mkdir html
./genhtml $2 $3
#echo x11 stress ...
./latencytest $1 $2 $3 $4 latency.out
@@ -19,7 +22,6 @@
sync
sleep 4
-
#echo disk write stress ..
./latencytest $1 $2 $3 $4 latency.out
./stress_diskwrite $5
@@ -27,7 +29,7 @@
sleep 1
cat latency.out
mv out.png html/$2x$3-diskwrite.png
-ls -la tmpfile*
+ls -la $latdiskroot/tmpfile*
sync
sleep 4
@@ -38,7 +40,7 @@
sleep 1
cat latency.out
mv out.png html/$2x$3-diskcopy.png
-ls -la tmpfile*
+ls -la $latdiskroot/tmpfile*
sync
sleep 4
@@ -48,8 +50,18 @@
killall latencytest 2/dev/null
sleep 1
cat latency.out
-ls -la tmpfile*
+ls -la $latdiskroot/tmpfile*
mv out.png html/$2x$3-diskread.png
-rm -f tmpfile tmpfile2
+#echo disk rm stress ...
+./latencytest $1 $2 $3 $4 latency.out
+sleep 1
+rm -f $latdiskroot/tmpfile $latdiskroot/tmpfile2
+sleep 1
+sync; sync; sync;
+sleep 20
+killall latencytest 2/dev/null
+sleep 1
+cat latency.out
+mv out.png html/$2x$3-diskrm.png
diff -Naur latencytest0.42-png/latencytest.c latencytest0.42-png+rl/latencytest.c
--- latencytest0.42-png/latencytest.c 2000-07-25 11:37:58.0 +0200
+++ latencytest0.42-png+rl/latencytest.c 2004-06-03 01:05:58.0 +0200
@@ -488,7 +488,7 @@
* set the process to realtime privs
*/
memset(schp, 0, sizeof(schp));
- schp.sched_priority = sched_get_priority_max(SCHED_FIFO);
+ schp.sched_priority = sched_get_priority_min(SCHED_FIFO)+1;
if (sched_setscheduler(0, SCHED_FIFO, schp) != 0) {
perror(sched_setscheduler);
diff -Naur latencytest0.42-png/stress_diskcopy latencytest0.42-png+rl/stress_diskcopy
--- latencytest0.42-png/stress_diskcopy 1999-06-04 01:29:42.0 +0200
+++ latencytest0.42-png+rl/stress_diskcopy 2004-06-03 00:38:33.0 +0200
@@ -1 +1 @@
-cp tmpfile tmpfile2
+cp $latdiskroot/tmpfile $latdiskroot/tmpfile2
diff -Naur latencytest0.42-png/stress_diskread latencytest0.42-png+rl/stress_diskread
--- latencytest0.42-png/stress_diskread 1999-06-05 17:09:21.0 +0200
+++ latencytest0.42-png+rl/stress_diskread 2004-06-03 00:38
Re: [linux-audio-dev] Re: mini Review: Re: [Jackit-devel] latest CVS commit
On Tuesday 02 December 2003 05.16, Paul Davis wrote:
On Monday 01 December 2003 20.48, Paul Davis wrote:
i'd appreciate test reports ASAP, so that out trusty release
technician (the very wonderful taybin rutkin) can get a new release
out in the near future.
alsa_driver.c
driver-period_usecs =3D
(jack_time_t) floor float) driver-frames_per_cycle) /
driver-frame_rate) * 100.0f);
jackd/engine.c
poll_timeout =3D (engine-control-real_time =3D=3D 0 ?
engine-client_timeout_msecs :
engine-driver-period_usecs/1000);
- - -
if (poll (pfd, 1, poll_timeout) 0) {
Isn't this dangerous?
* What happens if period_usecs 1000?
Well poll_timeout gets 0 since period_usecs is an uint64_t
A poll with time out of zero will return immediately.
OK?
well, there is no alternative really. the kernel has no time
resolution less than 1msec even on modern kernels with HZ=1000. so
setting it lower could be rather risky, given the use of busywaits for
some other SCHED_FIFO delays (see the code for usleep).
but yes, you are right, it is dangerous. we should probably sleep for
max (1msec, period_msecs). agree?
Either that or (period_msecs + 1)
You should get a solid lockup if your period_msecs is lower than 1000 us ==
1 ms when running SCHED_FIFO... (unless you run clients at the same
priority and the kernel yields at poll - I don't think it does)
/RogerL
--
Roger Larsson
Skellefteå
Sweden
jack context switches (Was: Re: [linux-audio-dev] another kernel patch?)
On Tuesday 02 December 2003 02.50, Jack O'Quin wrote: Roger Larsson [EMAIL PROTECTED] writes: But it is still a way to see that no client burns cycles where it should not - jackd would not start (or stop). And you _can_ get fewer context switches, but only if some client burns extra cycles. Compare: With jackd as highest priority: Context switches: A-jd [start B] -A [tail] -B-jd With jackd as lowest priority: Context switches: A-jd-B-jd The actual JACK process cycle works like this: jd-A-B-jd How? How can B start immediately after A without ANY jackd running in between? (I am talking about external clients) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] another kernel patch?
On Sunday 30 November 2003 05.03, Jack O'Quin wrote: Roger Larsson [EMAIL PROTECTED] writes: That's right. But, Paul and I have been working closely with this and don't have much faith in the correctness of the 2.4 scheduler. Have you told kernel developers about this? This can be rather critical in embedded systems. No. It's rather difficult to prove. There's no smoking gun. But, I have no reason to believe that it works correctly, and I suspect that it probably does not. Is there a tread documenting your problems? Anything else? What problem does this solve? It will better match what you think you do. When jackd writes in the FIFO - expect client to start. Jackd waits for baton - wait for client to finish. But with jackd as highest priority: Jackd writes to FIFO - nothing will happen for client. Jackd _waits_ for baton - now client start, processes, finishes. I don't see why this is a problem. All this has to happen for every cycle, anyway. What difference does it make? And, why do you think we don't understand this? Sorry, I got my wording wrong! (you = developers in general) But it is still a way to see that no client burns cycles where it should not - jackd would not start (or stop). And you _can_ get fewer context switches, but only if some client burns extra cycles. Compare: With jackd as highest priority: * Client A running * Jackd starts (higher prio than A), telling client B to run * Jackd waits * Client A contiues to run (higher prio than B) - this could be something very short like preparing to go to sleep after passing the baton. * Client A done, goes to sleep. * Client B runs - but later than expected... * Client B done * Jackd continues Context switches: A-jd-A-B-jd With jackd as lowest priority: * Client A runs * Jackd wants to start but has to wait * Client A continues to run * Client A done, goes to sleep * Jackd starts, telling client B to run * Client B starts * Client B done * Jackd continues, waits - B is finished Context switches: A-jd-B-jd Trying with jackd with lowest priority would atleast give you another data point. /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] mini Review: Re: [Jackit-devel] latest CVS commit
On Monday 01 December 2003 20.48, Paul Davis wrote:
i'd appreciate test reports ASAP, so that out trusty release
technician (the very wonderful taybin rutkin) can get a new release
out in the near future.
alsa_driver.c
driver-period_usecs =
(jack_time_t) floor float) driver-frames_per_cycle) /
driver-frame_rate) * 100.0f);
jackd/engine.c
poll_timeout = (engine-control-real_time == 0 ?
engine-client_timeout_msecs :
engine-driver-period_usecs/1000);
- - -
if (poll (pfd, 1, poll_timeout) 0) {
Isn't this dangerous?
* What happens if period_usecs 1000?
Well poll_timeout gets 0 since period_usecs is an uint64_t
A poll with time out of zero will return immediately.
OK?
/RogerL
--
Roger Larsson
Skellefteå
Sweden
Re: [linux-audio-dev] another kernel patch?
On Saturday 29 November 2003 04.54, Paul Davis wrote: while we're discussing various kernel security patches to facilitate easier access to SCHED_FIFO/mlockall, i have another idea for a patch that some people *might* like. a new system call. call it switch_to(). takes a PID (actually, it needs some kind of TID), and does something very similar to sched_yield() except instead of giving up the processor to whatever the scheduler thinks is right, it yields to the specific process/thread. security: the target thread has to be using the same RT scheduling policy (FIFO or RR) as the initiating thread. this means that it can only be used to cause denial-of-service attacks that were already trivial (because SCHED_FIFO was already available to the initiating thread). this could be used to completely short-circuit the FIFO mechanism used by JACK in favor of completely deterministic, FS-lock-free system. when you add in stuff like this (from ingo, discussing NPTL): our kernel thread context switch latency is below 1 usec on a typical P4 box, so our NPT library should compare pretty favorably even in such benchmarks. We get from the pthread_create() call to the first user instruction of the specified thread-function code in less than 2 usecs, and we get from pthread_exit() to the thread that does the pthread_join() in less than 2 usecs as well - all of these operations are done via a single system-call and a single context switch. you end up with a truly superb architecture for the kind of thing we're doing with JACK already. The new scheduler is very good at switching processes quickly. The old one had to walk throu all to decide which one to take, not the new one. So you will not save much by telling wich process to switch to. I think that the above figures include the scheduler! (Have to check) however, note this comment from ingo as well, which i consider short-sighted, and is part of the reason for my thinking about switch_to(): M:N's big mistake is that it concentrates on what matters the least: useruser context switches. Nothing really wants to do that. And if it does, it's contended on some userspace locking object, at which point it doesnt really matter whether the cost of switching is 1 usec or 0.5 usecs, the main application cost is the lost paralellism and increased cache trashing due to the serialization - independently of what kind of threading abstraction is used. This was about switching threads within a single process - simulating threads in a system where they are not handled by the kernel. Not about switching from one user process to another. any thoughts? adding a syscall is a pretty trivial patch to create. And then the hard part begins - trying to get it accepted by Linus :-) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] another kernel patch?
On Saturday 29 November 2003 21.55, Paul Davis wrote: No scheduler that I've ever seen, written, or heard of is going to accept JACK's opinion on this subject as anything more than a strong hint. The scheduler is the main loop of the entire system, *it* decides what to run next. i'm not trying to subvert the general operation of the scheduler. it will still intervene when it needs/wants to. Let see - what you want is this: what i want to stop is this: jackd is about running the process cycle, and wants to hand control to the next client. it writes to a FIFO to wakeup the next client, If next client is a RT process (SCHED_FIFO/RR) then it WILL be selected unless there is another RT process with higher priority, and lower than jackd otherwice it would have been running in the first place. and goes to sleep waiting for the baton-passing to wake it up again. Jackd waits... however, for some bizarre reason, the scheduler does not in fact run the next client, but does something else instead. If this ever happens (and something else is not within in the priority range new process ... jackd) then it is a scheduler BUG - and should be fixed! Cases where RT processes ends up flushing page cache to disk should be investigated and fixed. If any... put another way, i want JACK-the-system to keep running smoothly, even though this includes crossing process boundaries. i don't want the handoff of control that is implicit in running the process cycle to return control to the scheduler unnaturally. But the current scheme can do more. Suppose you are running on a SMP machine (Chip level SMP processors are getting common - there is even a Xilinx FPGA - Virtex Pro with four PowerPCs [Not G5 but anyway...]!) Suppose that in this situation you have made data available for several processes - how can you start them all with your suggested interface? (You might want to start them all and let the scheduler figure out what to do - hyperthreading is nice when using the same data) With the current scheme (OK, I am not quite up to date) you could write to the FIFOs of all ready processes before starting to wait on the baton. [giving the sub processes different priority levels will order them in case you have less processors than CPUs] Suggestions are welcome, but JACK truly does *not* know for certain what to run next. An interrupt could intervene and make a higher-priority realtime process ready. Or, some other processor could unlock such a thread. At the very least, there probably needs to be code in the return path from the system call like the old Unix logic of testing `runrun' and calling sched(). obviously, all these would remain in place and we would be doing nothing subvert them. all we are doing is avoiding a schedule() caused by jackd blocking-on-read and the next client being marked runnable. both jackd and the next client should be considered part of the same system. To get this you should run the clients with higher priority than jackd! Then they would be started and run before returning from the FIFO write! :-) client A requests data from FIFO kernel lets A waits on FIFO jackd writes data to FIFO kernel puts data on FIFO kernel wakes up waiters --- client A has higher prio than jackd, it will preempt jackd while --- jackd is still in the kernel. client A wakes up processes returns baton waits for more data on FIFO kernel finds next task to run and finds jackd (still on run queue) kernel returns jackd checks for baton - got it. jackd determines what client to run next. Maybe jack should handle the priorities for jack clients - priorities is a relative issue anyway... Then jackd could use this scheme when there is only one client to run - and the lower priority clients when there are several. /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] another kernel patch?
On Sunday 30 November 2003 03.19, Jack O'Quin wrote: Roger Larsson [EMAIL PROTECTED] writes: If next client is a RT process (SCHED_FIFO/RR) then it WILL be selected unless there is another RT process with higher priority, and lower than jackd otherwice it would have been running in the first place. I wish this were the true. But, I strongly suspect that it is not. however, for some bizarre reason, the scheduler does not in fact run the next client, but does something else instead. If this ever happens (and something else is not within in the priority range new process ... jackd) then it is a scheduler BUG - and should be fixed! That's right. But, Paul and I have been working closely with this and don't have much faith in the correctness of the 2.4 scheduler. Have you told kernel developers about this? This can be rather critical in embedded systems. Like all non-trivial software components it has bugs, and getting them fixed is difficult, if not impossible. But the scheduler is VERY simple when you are using RT (SCHED_FIFO/RR). It only selects the one with highest priority. So unless your thread falls off the run queue - critical. Does not timely get on the run queue after wake up - critical. Gets stuck somewhere else - program bug? But it could also be jackd the gets stuck doing something else... (this is the reason why I am not comfortable with the printfs in my rt_monitor - they use the console and it is bad...) Anyway try to describe your problems in linux-kernel (cc me) With the current scheme (OK, I am not quite up to date) you could write to the FIFOs of all ready processes before starting to wait on the baton. [giving the sub processes different priority levels will order them in case you have less processors than CPUs] The current JACK implementation is completely single-threaded regardless of how many real or virtual processors your system possesses. But, for some (relatively complex) graphs a multithread schedule would be possible and desirable. So, theoretically your point is valid. Maybe jack should handle the priorities for jack clients - priorities is a relative issue anyway... Then jackd could use this scheme when there is only one client to run - and the lower priority clients when there are several. What problem does this solve? It will better match what you think you do. When jackd writes in the FIFO - expect client to start. Jackd waits for baton - wait for client to finish. But with jackd as highest priority: Jackd writes to FIFO - nothing will happen for client. Jackd _waits_ for baton - now client start, processes, finishes. You could try a low priority jackd (still RT) in this way to see if things behave better. Next week I will read some jack and kernel code if you can try to explain what happens. /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Tracker
On Friday 28 November 2003 21.27, Paul Davis wrote: as paul says, opt/ is a fine solution for this, as everything just symlinks, but it's sad that it never took off. i don't think its a fine solution. i never saw anything useful about /opt at all. it just created a *third* possible install location. The alternative for distributions is to put big packages with lots of executables under /usr since /usr/local is mine (or the local administrators) But executables should really end up in /usr/bin, non architecture specific files in /usr/share - but then you need to handle several versions too... In this situation I do not mind /opt ... /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: POSIX caps/realtime/root processes
On Thursday 27 November 2003 13.42, Kjetil Svalastog Matheussen wrote: On Thu, 27 Nov 2003, Martijn Sipkema wrote: [...] So how is the low-latency situation for 2.6? I did install 2.6 on my private machine, but was not able to get better performance than 2.4 with ll+pre (kicked out of jack-graph pretty soon with 128 frames period). Is there a trick to get better lowlatency performance with 2.6 I don't know about? Did you compile with CONFIG_PREEMPT enabled? Yes. :) echo 1 /proc/sys/kernel/lowlatency ? If you have enabled Control low latency with sysctl It might be possible that something like lock break is needed... Lots of good stuff is available at: http://www.tech9.net/rml/linux/ But most are for 2.4 or early 2.5 kernels /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Linux Audio article in DATOR MAGAZIN 2003-12 (swedish)!
7 pages with written by Albert Veli Take a look at the front page :-) http://www.datormagazin.se/ssjs/aktuellt_nr.html It mentions, with varying levels of detail: * Alsa (three pages) * GNOME ALSA Mixer (with screen dump? - it look old and boring compared to KMix in KDE 3.2 IMHO) * Jack * Qjackctl (with screen capture?) * Fluidsynt * Timidity++ * LADSPA * Jack-rack * Ecamegapedal (with screen capture) * Rosegarden-4 (with screen capture) * ReZound (with screen capture) * Muse * Audacity * Ardour [problematic to compile, steep learning curve the article suggests looking at www.djcj.org/LAU/quicktoots * Ecasound * AlsaModularSynth * Legasynth * SpiralSynth * Pd * Soundtracker, cheesetracker * Alsaplayer * xmms /RogerL -- Roger Larsson Skellefteå Sweden
[CODE] rts with memory protection (Was: Re: [linux-audio-dev] Re: linux-audio-dev Digest, Vol 2, Issue 24)
On Wednesday 19 November 2003 15.53, Jack O'Quin wrote: Roger Larsson [EMAIL PROTECTED] writes: I have an idea! rt_monitor is running mlockall and I avoided * dynamic memory allocation * system calls - If I split the function into server and monitor (two threads) it would be even better. So rt_monitor itself should never get stuck requesting more memory, nor get stuck in a system call getting stuck on another process requesting more memory. Sounds like a good idea. Something like this... some build and test bugs corrected. /RogerL -- Roger Larsson Skellefteå Sweden rts.zip Description: Zip archive
Re: [linux-audio-dev] Re: linux-audio-dev Digest, Vol 2, Issue 24
On Wednesday 19 November 2003 01.40, Jack O'Quin wrote: Roger Larsson [EMAIL PROTECTED] writes: Problem is - why doesn't most distributions even ship with wrappers suid to be able to start the application with SCHED_FIFO/RT/mlock? - It is due to risks of local Denial Of Service attacks (intentional or unintentional) That seems logical, but AFAICT the actual reason is because of a security hole introduced by CAP_SETPCAP (which was not part of the original POSIX draft spec). Before this screwup was detected, kernels shipped with capabilities enabled. If an attacker manages to subvert one of the system daemons with a buffer overflow (sendmail is a frequent target), it can use CAP_SETPCAP to deny capabilities to other system services that need them to perform their jobs, including monitoring system security. Yes, lets take a specific example arts (KDE audio) and artswrapper. Artswrapper should be suid root to give arts the desired SCHED_FIFO, that is close to the full function of artswrapper - but still it is not shipped suid root... So with any scheme that opens up these holes you have to deliver a way to protect from the downsides. Clearly this is desirable. But, for many audio workstations it is *not* mandatory. But on those audio workstations you have NO problem to make wrappers suid root. * The audio workstation case can always be handled. * The problematic case is common desktop where users get a much worse experience than they should - and it might scare people off. My monitor protects from CPU overuse, but what about memory? How to protect from an application that mlockall(MCL_FUTURE) and has a memory leak? If you fail to solve this problem, then we end up back where we are right now: patching kernels or running untrusted audio applications as root. This solution is much worse than the problem you were trying to solve. I have an idea! rt_monitor is running mlockall and I avoided * dynamic memory allocation * system calls - If I split the function into server and monitor (two threads) it would be even better. So rt_monitor itself should never get stuck requesting more memory, nor get stuck in a system call getting stuck on another process requesting more memory. But it could monitor the RT processes memory usage and kill those crossing the line. Will try it tonight... /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: linux-audio-dev Digest, Vol 2, Issue 24
On Tuesday 18 November 2003 02.41, Jack O'Quin wrote: Fernando Pablo Lopez-Lezcano [EMAIL PROTECTED] writes: - Can not provide mlockall since it has no pid parameter - the monitor can't do it for another process. (I would say that it is unlikely that pages used in a tight audio loop would be thrown out - big buffers might... Add additional page reads?) Well, I'd say this is the showstopper. We really need this. Unlikely is not enough. Eventually memory will run out and the wrong page will fault and we get a click. We have to be able to lock memory.. Agreed. IMHO, mlock() is mandatory, certainly for JACK applications. Problem is - why doesn't most distributions even ship with wrappers suid to be able to start the application with SCHED_FIFO/RT/mlock? - It is due to risks of local Denial Of Service attacks (intentional or unintentional) So with any scheme that opens up these holes you have to deliver a way to protect from the downsides. My monitor protects from CPU overuse, but what about memory? How to protect from an application that mlockall(MCL_FUTURE) and has a memory leak? One important thing to remember - if you like to get broad acceptance you have to suggest a solution that solves these problems. I would say that the rt_monitor or some other means to do the same thing is mandatory to get that kind of acceptance. The big difference between realtime and most other kinds of performance work is that it focuses on tuning the worst case, not the average. Paging works fine on average, but in the worst case your recording session gets blown. SCHED_FIFO does not make ANY guarantees on worst case! Otherwise, a good solution. Perhaps adequate for some applications. But at the same time SCHED_FIFO is adequate for most applications. See my point? :-) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: linux-audio-dev Digest, Vol 2, Issue 24
On Tuesday 18 November 2003 13.28, Stefan Nitschke wrote: BUT, I think a userspace daemon that starts at boot time and protects against lockups (rt_monitor) would be a very good thing to have. Yes indeed, but on my XP machine which freezes after a few seconds after a client had connected to jack even rt_monitor did not helped, the machine keeped totally locked. Ping is still answered but thats it. ping is answered = IRQs live. Could you please try this. Start rt_monitor in a text console CTRL-ALT-F1 Start your client, return to console - what does rt_monitor print? (It could be the memory leak and mlockall(FUTURE) problem, As I do not check that yet. Another alternative is a fast forking client - they are hard to stop) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: linux-audio-dev Digest, Vol 2, Issue 24
On Tuesday 18 November 2003 20.57, Stefan Nitschke wrote: ping is answered = IRQs live. Could you please try this. Start rt_monitor in a text console CTRL-ALT-F1 Start your client, return to console - what does rt_monitor print? (It could be the memory leak and mlockall(FUTURE) problem, As I do not check that yet. Another alternative is a fast forking client - they are hard to stop) OK, I tested it. There is no output at all from rt_monitor. I used the following: - starting rt_monitor in a console as root - ssh login from remote machine - starting jack - starting a jack client What client? Any client? after a few seconds the machine was froozen and no output from rt_monitor. Hmm... I assume its a hardware bug. On my Inspiron 8500 with nearly the same linux-2.4.19.SuSE kernel (plus patch for speed-step) I dont have any freezes at all. But you do have it recompiled for the right CPU architecture and not only moved? BTW: There is a prefech bug on AMD processors - try to run it recompiled with less optimization, newer kernels have workarounds. Another thing to try is to run memtest86 (SuSE has it in Lilo menu) let it run for a while (full night) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: linux-audio-dev Digest, Vol 2, Issue 24
On Monday 17 November 2003 06.17, Fernando Pablo Lopez-Lezcano wrote: On Sun, 2003-11-16 at 08:02, Paul Davis wrote: I've been thinking about ways to use this feature to improve and simplify the current security situation for Linux audio. No conclusions, but here are some thoughts for discussion: (1) There should a simple way for the sysadmin to reliably disallow [ .. ] (2) Using sysctl, set a group id (like `audio') for which realtime [ ... ] (3) We could also define a default realtime group (gid 0 maybe), What about this one: (4) Let the user that is currently physical logged in to the machine get realtime privileges. i'd be interested to hear from fernando about this kind of thing. many of us on LAD work on what are to all effects and purposes single user machines. i'd like to hear how policies like 1-4 above, or others, appear in the context of an academic shared resource environment. [academic is probably irrelevant, a commercial studio should see the advantages of the security model if educated] As far as I understand these are the current options: a) capabilities b) simple sysctl patch to the kernel (like the one that Kjetil posted) c) security module, with possible additional control through sysctl What about: d) redefining sched_setscheduler using LD_PRELOAD with running rt_monitor + No change in kernel. + No change in application code. (Like capabilities - do not check uid, but it can fake uids too!) + Can use whatever kernel provides + Only requires the rt_monitor to be started by root - can be done by init + Protects the system from overuse (and lockups) due to bugs in code. - Can not provide mlockall since it has no pid parameter - the monitor can't do it for another process. (I would say that it is unlikely that pages used in a tight audio loop would be thrown out - big buffers might... Add additional page reads?) /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] [CODE RFC] redefining sched_setscheduler using LD_PRELOAD
Hi, (Jack O'Quin got me thinking about rt_monitor again - thanks!) This is an alternative to capabilities / SCHED_SOFTRR... I use my old rt_monitor * protects the system on RT overload * performs the actual setting of scheduler type * it does not even have to be visible for lusers! My old testprogram RT has been converted to a library fit for preloading. It works like this. As root start # rt_monitor As normal user start latencytest $ ./rts path/to/latencytest none 2 128 10 rts has two options -i do not raise the priority. -v be more verbose. * It does also limit the effective min/max range. Do also try (don't try this as root unless rt_monitor is running) $ ./rts path/to/latencytest none 2 1024 99 * In this case rt_monitor captures the process and renices it. (SCHED_OTHER + nice) BUGS: * better build and installation system * could ignore RT processes already running when rt_monitor starts * rt_monitor functionallity has not been reverified... * it can't lock memory :-( No PIDs in calls. * code that checks uid before using the redefined functions won't work (I have an modified latencytest) * client get stuck if there is no monitor running. /RogerL -- Roger Larsson Skellefteå Sweden rts.zip Description: Zip archive
[linux-audio-dev] Fwd: [[NO] PATCH] cfq + io priorities
Hi, I guess this will be of interest to some of you... An even newer version has shown up on linux-kernel. /RogerL From: Jens Axboe Hi, I've implemented IO nice levels in the CFQ io scheduler. It works as follows. A process has an assigned io nice level, anywhere from 0 to 20. Both of these end values are special - 0 means the process is only allowed to do io if the disk is idle, and 20 means the process io is considered realtime. Realtime IO always gets first access to the disk. Values from 1 to 19 assign 5-95% of disk bandwidth to that process. Any io class is allowed to use all of disk bandwidth in absence of higher priority io. Idle and realtime IO settings work as expected, but not much tuning has gone into making sure that the individual levels in-between work 100% as expected. It should be good enough for some testing at least, even if it has some holes. About the patch: stuff like this really needs some resource management abstraction like CKRM. Right now we just look at the tgid of the process. I've added two syscalls for setting and getting io priority. Don't consider this final or anything, it's just easy for testing. Patch has been tested on x86 and ppc, syscalls are also added for x86_64. I'm attaching the simple ionice tool. It's used as follows: # ionice -n20 bash starts a bash shell with realtime io. Beware that io level is inherited on fork, so any program you start from this shell will also run with realtime io. # ionice -n0 dbench 32 run some dbench thrasher, but only when disk is idle. Pretty straight forward :-) For really good results, you probably also want to set cpu nice level. Needless to say, a realtime io process can only submit io when it gets scheduled. Default IO priority for a new process is 10. Patch is against bk-current. -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] StreamIt - compiler for modern streaming systems
(resent, previously sent from non member account) Found this on comp.arch StreamIt is a programming language and a compilation infrastructure, specifically engineered for modern streaming systems. It is designed to facilitate the programming of large streaming applications, as well as their efficient and effective mapping to a wide variety of target architectures, including commercial-off-the-shelf uniprocessors, multi-clustered architectures, and the emerging class of grid-processors. http://cag.csail.mit.edu/streamit/ It does handle feedback loops among other things. /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Fixed vs Floating point comparison.
(resent, previously sent from non member account)
First: Is the code correct?
pos=lrintf(counter);
Shouldn't pos be lower than counter and pos+1 higher? Like this
pos = counter pos+1
pos = (int)(counter);
= I use this in my tests.
Second: Why the 'volatile'?
You should never be that timing sensitive that it makes a difference...
(let the producer enter an interrupt and...)
Third: Compilators hates pointers and loves arrays - strange ehh..
Modified the code to this, note the added loop counter.
int ix;
for (ix = 0; ix todo; ix++) {
pos=(int)(counter);
counter+=advance_f;
res = src_ptr[pos];
res_next = src_ptr[pos+1];
res += (res_next-res) * (counter-(float)pos);
res *= volume_f;
dst_ptr[ix] +=res;
}
Results:
PIII 933 MHzvolatilenon volatile
pointer dst 5.0 s 4.9 s
array dst 4.5 s 4.1 s
/RogerL
--
Roger Larsson
Skellefteå
Sweden
Re: [linux-audio-dev] Linux 2.6 not a latency panacea?
On Thursday 14 August 2003 00.09, Joshua Haberman wrote: I am distressed. It was my understanding that the 2.5/2.6 kernel branch was undergoing significant scheduler and latency work, and that 2.6 would eliminate the kernel from the list of obstacles of low-latency on Linux. It will have the preemptable kernel patch, the new scheduler, and all of Ingo Molnar's low-latency work. Claims were being thrown around that 2.6 would be the lowest-latency operating system on the planet. So how is it that we're in the 2.6.0-test series and people are complaining about audio skipping in **XMMS**, which uses three second buffers by default?? If people are getting skips from high-latency playback, what hope is there for low-latency audio? A series of patches are coming from both Ingo and Con Kolivas attempting to address this, but the fact they are just now throwing around potential solutions erodes at my faith that they really understand the problem or how to solve it. Is 2.6.x going to be suitable for low-latency (or even reliable high-latency) audio? Or is it going to be more of the same: patching the kernel, tweaking parameters, reading magical incantations, and hoping for the best? Reassure me please! Josh This is when running the default scheduler (as non root / non suid root). Then you are not allowed to use SCHED_FIFO nor SCHED_RR The problem such a process might run into is if it needs service or a resource held by a blocked process... BTW There have been discussions about a new scheduling class SCHED_SOFTRR. It would be available for all users. But the total usage would be limited. If SCHED_SOFTRR were overused those processes would run out of their timeslice (SCHED_RR never runs out of their timeslice) I think this feature would be pretty cool! And adding this for latency sensitive bandwith limited streaming applications could simplify lots of stuff for the default scheduler... /RogerL -- Roger Larsson Skellefteå Sweden
Monitor - again Re: [linux-audio-dev] Linux 2.6 not a latency panacea?
On Thursday 14 August 2003 15.59, Takashi Iwai wrote: At Thu, 14 Aug 2003 02:55:28 +0300 (EEST), Kai Vehmanen wrote: For example, one new approach to the problem SCHED_SOFTRR, see: http://www.xmailserver.org/linux-patches/softrr.html http://www.ussg.iu.edu/hypermail/linux/kernel/0307.1/1729.html It's unlikely to get something like this enabled by default in the vanilla kernel, but we might be able to get a kernel option (no patching!). But, but, as you can see from the discussion, they are talking about totally different things (how XMMS/realplayer performs)... ... basicly a way to get benefits of SCHED_FIFO but without need for root privileges. Now we just need to push these to the standard kernel somehow. i think the most benifit of soft-RR is that it doesn't bring your box hanging up even if a RT-process gets into an infinite loop. this will help to sort out the problem when a JACK system freezes with SCHED_FIFO. (i.e. if it happens with soft-RR, it's a kernel/driver bug :) My monitor can do this too. But you need to have that running. It degrades ALL SCHED_FIFO/RT processes to SCHED_OTHER on overuse - processes that do run as root could be protected. (FUTURE) running by the normal user is an additional gift, for my eyes. such a feature can be implemented with a wrapper (library), too, as jackstart does. You can request SCHED_FIFO/RT from my monitor. Current implementation is working but it can be improved! It uses static allocations intentionally. The problem is to get it accepted - should it be added to jack(d)? /RogerL -- Roger Larsson Skellefteå Sweden rt_monitor.zip Description: Zip archive
Re: [linux-audio-dev] [OT] [for sale] RME Hammerfall (digi9652)
Hmm... I guess Paul can throw in an signed(!) linux driver CD. Let the bidding start :-) /RogerL On Tuesday 04 March 2003 23:13, Sebastien Metrot wrote: Huh, no linux driver CD? I'm not interested! (ok, easy one, sorry :-). Sebastien PS: I allready own one, great stuff, don't hesitate. - Original Message - From: Paul Davis [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, 04 March, 2003 22:47 Subject: [linux-audio-dev] [OT] [for sale] RME Hammerfall (digi9652) (sorry for the off-topic post, but i want people on these lists to have the first chance at this) Rev 1.5 Hammerfall Digi9652 all cables Windows/Mac driver CD original packing materials current street price: about $580 asking price: $480 OBO + shipping -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Why? call back API and other thoughts...
On Thursday 27 February 2003 20:25, Tim Jansen wrote: Did the other platforms use a callback-driven approach because it is a superior API, or because it is the only way to have sound on cooperative-multitasking OSes like MacOS X and early Windows versions? Callback-driven APIs are much harder to use, especially with many existing codecs and frameworks that have been written with a push API in mind. I try to list some pro / con for call back and push APIs: push API: Pro: * simple to play a single file, basically a copy from source file to destination. Con: * hard to write plugins that way. Take a look at arts plugins. They all have a 'calculateBlock' = call back! Why? In a pure push model each processing step reads data from one file/pipe/device, processes it, pushes it to a file/pipe/device You get: * lots of threads/processes that are not optimally synchronized. Any thread is runnable when there are input available until the output is full. But that is not the important case, concider the case when the last processing steps output is almost empty (if it gets empty you will hear a click). How to prioritize it higher than all other threads? Should it always be higher? Suppose it is a mixer that has several inputs... Could be done by a super server that sets priorities depending on position in the line? This is not easy... * If plugins, with callback model, are used to build the application. Does it not make sense to build the whole application (audio part) the same way? There are some neat tricks that can be used, since the wrapper library can know where the destination is. * if the destination is in the same process, your output will end up somewere in your process memory. * On the other hand, suppose the destination is another application, it can allocate shared memory and let the output of your pluggin end up there. * If the output is destined to go to an audio board It could then give you a memory mapped hardware buffer instead of ordinary memory to avoid the final copy. (you will get different buffers on each process...) * if your output type does not match the input type of the destination, the library could automatically insert a converter/resampler either on your side or on the destination side (pick the one that gives less communication). * Can the destination change during the run? 1. Your application starts alone, output format matches one supported by hardware. = hardware buffers 2. Another application starts (suppose the device can have several destinations open at once - like SB Live!) = no change for your pluggin (but assume the format of this pluggin is not supported by hardware = in process buffer + automatic inserted convertion pluggin + hardware buffer) 3. Even more applications start... No more possible to output direct to hardware for all... suppose the library checks for matching data types - and the first application match perfectly! = the new application will get shared memory, your application will be changed to ordinary memory, these buffers will be mixed by an automatically inserted pluggin that outputs to the hardware buffer... 4. The new application ends. = hardware buffers again = your application/pluggin does not need to care. [No framework that I know of implements this today - especially not the automatic parts] With the push model your application needs to know about possible destinations, or treat everything as a file or shared memory. But how to handle the dynamic changes in the push model? Pipes, shared memory? It could also use a library that directs the produced buffer in the right direction (CSL?) - but it will be hard to eliminate extra copying. Note: Arts and the KDE multimedia framework does a lot of things right today. It even goes one step furter since it moves the pluggins, including input and output into one process - artsd. But currently it does not work perfectly together with applications using other frameworks. /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Fwd: CSL Motivation
Hi, There are discussions on kde-multimedia about the future of Linux/Unix multimedia (especially sound). This is one of the most interesting messages. /RogerL -- Forwarded Message -- Subject: CSL Motivation Date: Tuesday 25 February 2003 16:26 From: Tim Janik [EMAIL PROTECTED] To: KDE Multimedia [EMAIL PROTECTED] hey all, due to the recent discussions involving CSL (amongst other projects) on this list, i've put up a preliminary web presence at: http://sfk.evilplan.org/csl/ and, more importantly, stefan and i wrote a paper going into the details of why we think a project like CSL is necessary and what we intend to achieve with it: http://sfk.evilplan.org/csl/csl-paper.ps --- ciaoTJ ___ kde-multimedia mailing list [EMAIL PROTECTED] http://mail.kde.org/mailman/listinfo/kde-multimedia --- -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: Steve Harris' C++ v C benchmark
Some quick note: Testing that i have done suggest that you should ALWAYS define the architecture of your target. (I have not checked if the example does this but it is usually forgot...) -march=pentium3 or if you need it to run on older computers -mcpu=pentium3 (see man gcc search for Intel 386) And at least use optimization level -O1 (use -O3 to get automatic inlining) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] MIDI Clock and ALSA seq
On Friday 14 February 2003 16:18, Steve Harris wrote: On Fri, Feb 14, 2003 at 03:54:35 +0100, Roger Larsson wrote: On Friday 14 February 2003 15:02, Steve Harris wrote: Hi all, I'm trying to hack up a quick app that sends MIDI clock pulses in sync with a ringbuffer playback. I've got it reading MIDI (thanks to Matthias' great exmaple code), but I cant figure out how to send MIDI clock pulses explicitly. The ALSA seq interface lets me set it up to go automatically, but I want to send a clock pulse every 1/24th of the buffer, so the sync is correct. How about dividing the buffer in 24 frames. And send one MIDI event each frame? Doable? That's exactly what I'm trying to do, but I can't figure out how to construct the midi clock event. Run with 24 frames and a frame size of (needed buffer / 24). Run this with sched fifo, etc... Output 24 frames of silence. Write some more silent frames (it wont be accepted before one is free), send MIDI event for each. [Gives the MIDI device time to sync] Write a frame of real data. Send MIDI event. Repeat. This would give you some jitter but the MIDI device should filter that. /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] binary package of ardour - can be found in SuSE 8.1 (earlier?)
~ rpm -q ardour-cvs20020816-21 ardour-cvs20020816-21 roger@jeloin:~ man rpm Reformatting rpm(8), please wait... Still saving the page, please wait... roger@jeloin:~ rpm -q ardour-cvs20020816-21 -i Name: ardour Relocations: (not relocateable) Version : cvs20020816 Vendor: SuSE Linux AG, Nuernberg , Germany Release : 21Build Date: ons 11 sep 2002 00.36.09 Install date: fre 27 dec 2002 16.40.07 Build Host: Fatou.suse.de Group : Productivity/Multimedia/Sound/Editors and Convertors Source RPM: ardour-cvs20020816-21.src.rpm Size: 9069796 License: GPL Packager: http://www.suse.de/feedback URL : http://ardour.sourceforge.net/ Summary : Multitrack hardware recording system Description : Ardour is a multichannel hard disk recorder (HDR), rapidly evolving into a Digit al Audio Workstation (DAW). It is capable of simultaneously recording 24 or more channels of 32 bit audio at 48kHz. Ardour is intended to function as a professional HDR system, replacing dedicat ed hardware solutions such as the Mackie HDR, the Tascam 2424 and more traditional tape systems like t he Alesis ADAT series. It supports MIDI Machine Control and so can be controlled from any MMC controlle r, such as the Mackie Digital 8 Bus mixer and many other modern digital mixers. Authors: Paul Davis [EMAIL PROTECTED] Distribution: SuSE Linux 8.1 (i386) roger@jeloin:~ roger@jeloin:~ roger@jeloin:~ rpm -q ardour-cvs20020816-21 -i Name: ardour Relocations: (not relocateable) Version : cvs20020816 Vendor: SuSE Linux AG, Nuernberg, Germany Release : 21Build Date: ons 11 sep 2002 00.36.09 Install date: fre 27 dec 2002 16.40.07 Build Host: Fatou.suse.de Group : Productivity/Multimedia/Sound/Editors and Convertors Source RPM: ardour-cvs20020816-21.src.rpm Size: 9069796 License: GPL Packager: http://www.suse.de/feedback URL : http://ardour.sourceforge.net/ Summary : Multitrack hardware recording system Description : Ardour is a multichannel hard disk recorder (HDR), rapidly evolving into a Digital Audio Workstation (DAW). It is capable of simultaneously recording 24 or more channels of 32 bit audio at 48kHz. Ardour is intended to function as a professional HDR system, replacing dedicated hardware solutions such as the Mackie HDR, the Tascam 2424 and more traditional tape systems like the Alesis ADAT series. It supports MIDI Machine Control and so can be controlled from any MMC controller, such as the Mackie Digital 8 Bus mixer and many other modern digital mixers. Authors: Paul Davis [EMAIL PROTECTED] Distribution: SuSE Linux 8.1 (i386) -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Audio s/w ui swallowing
On Monday 09 December 2002 20:10, Steve Harris wrote: The world and his dog seems to be releasing macos/windows audio s/w that looks like 19 rack units. Anyone know enough about X to know if its possible to make X apps open thier main window inside a standard sized cabinet (ala Reason). I'm assuming it would be ok to require the app to be a certian size and have explicit support, but I guess it couldn't put any restrictions on toolkit. Other than looking cool, it would actually be a useful way to keep window clutter down. Not that we have any 19 lookalike apps yet, but I guess we will do at some point. - Steve Why not add another option to the window manager instead? (much like Always on top) Add to window group - Audio rack Let the window manager keep them with a common width and split height. Resize and Move all windows together. - Possible? /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] musicdsp
http://www.musicdsp.org/ Needs help with their Linux information... Any takers? /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
On Sunday 01 September 2002 18.50, Paul Davis wrote:
[I wrote:]
The only thing the sound servers need to be able to handle
RT audio are:
* one of the RT scheduling methods.
* locked memory. (Not protected at the moment - hard limits?)
This can be archived with a wrapper that drops ALL other priorities
before running the real application.
this is, alas, not true. to use RT scheduling, you need either root
uid or CAP_RESOURCE. if you have CAP_RESOURCE or root uid, you can do
many, many things (such as write to /dev/kmem) that can lead to a DOS.
So this code from artswrapper.c does not work then?
(hmm, shouldn't it should do setegid too...)
/* drop root privileges if running setuid root
(due to realtime priority stuff) */
if (geteuid() != getuid())
{
#if defined (HAVE_SETEUID) !defined (HAVE_SETEUID_FAKE)
seteuid(getuid());
#else
setreuid(-1, getuid());
#endif
}
But we can play safer than that!
Since the priorities can be set by another process, we can let the monitor do
it! (In the same way as it can reduce the priorities). And if we do it this
way we can build a higher level interface - requesting C cycles or MACs every
T microseconds. Or is ther a better interface?
Problems:
* it is hard to lock the memory... since mlockall does not work
over forks...
* How to prevent ANY program to request this? Might not be a problem since we
will not give away more cycles than available...
This might be a very good thing - giving the ability to request RT prio for
any program.
the central problem is that there is no way to tell the kernel this
thread should get to run whenever it wants, but do nothing else that a
normal process cannot do, and likewise, this vm address space can
lock down up to 128MB of memory. we don't have sufficiently
granularity to do this, so we have to grant overly broad access. that
in turns means that anything granted such access has many, many more
ways of attacking the system than burning CPU cycles or locking down
too much memory (and, BTW, i'm not sure if this latter issue is a
problem or not with the emergence of process killing logic in the
kernel these days).
The OOM killer might be fooled into killing an other random process...
(I have to check the code to be sure what will happen in this unnormal
situation)
My feeling is that this should be handled by the monitor aswell.
Maybe by sharing locked memory with the processes or by introducing
a new system call. Anyway - on a computer with enough memory this
should not be a problem. (but it might since several kernel versions might
swap out pages rather than flushing a recently read or written file...)
to make this situation better means recognizing the utility of
granting very specific resources to be used or more or less
exclusively by a single task, even when those resources more or less
mean all of the machine. this is not something that POSIX or Linux
provides at this time.
--p
I will give this monitor addition a try tonight.
/RogerL
--
Roger Larsson
Skellefteå
Sweden
[linux-audio-dev] [RFC SOURCE] rt_monitor take 2
With this monitor any process can request RT priorities.
If those (or other) processes overloads the system.
They will be returned to normal priorities.
Note:
* this code is still experimental. I had a complete
lockup because my busy system detection did not work...
(removed since it was not necessary)
* It does probably not work on SMP - I have not given it
much of a thought yet...
compile the source:
gcc -Wall rt.c -o rt
gcc -Wall rt_monitor.c -o rt_monitor
then as root:
mkfifo -m 622 /var/named/rt-request
./rt_monitor
start another shell (as a normal user - non root)
./rt
output from ./rt will look like this
policy 0- does not have RT priority
policy 0- request sent, waiting...
..
policy 1- got rt prio
- sleeps for 5 s
- lockup!!! (for about 4 s)
- rt_monitor reduces the prio, rt exits when detected
/RogerL
--
Roger Larsson
Skellefteå
Sweden
/* RT user.
Copyright (c) 2002 Roger Larsson [EMAIL PROTECTED]
This program is free software; you can redistribute it and/or
modify it under the terms of version 2 of the GNU General Public
License as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
Thanks to autor of KSysGuard Chris Schlaeger for borrowed code...
*/
#include sys/types.h
#include sched.h
#include stdio.h
#include unistd.h
#include errno.h
struct request
{
pid_t pid;
char _filler[32];
};
int main(int argc, char *argv[])
{
struct request request;
FILE *reqf;
reqf = fopen(/var/named/rt-request, w);
if (reqf == NULL) {
perror(fopen);
return errno;
}
printf(policy %d\n, sched_getscheduler(0));
request.pid = getpid();
fwrite(request, 32, 1, reqf);
fclose(reqf); // important!
printf(policy %d\n, sched_getscheduler(0));
while (sched_getscheduler(0) == 0) {
putchar('.');
sleep(1);
}
printf(\npolicy %d\n, sched_getscheduler(0));
// well behaved
sleep(5);
while (sched_getscheduler(0) != 0) {
// someone did listen to my request...
// assume monitor is running
}
return 0;
}
/* RT monitor.
Copyright (c) 2002 Roger Larsson [EMAIL PROTECTED]
This program is free software; you can redistribute it and/or
modify it under the terms of version 2 of the GNU General Public
License as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
Thanks to autor of KSysGuard Chris Schlaeger for borrowed code...
*/
#include sys/types.h
#include sched.h
#include stdio.h
#include dirent.h
#include sys/mman.h
#include sys/stat.h
#include fcntl.h
#include unistd.h
#include ctype.h
#include sys/time.h
#include stdlib.h
int set_normal_priority(pid_t pid);
int isRT(pid_t pid)
{
int sched_class = sched_getscheduler( pid);
if (sched_class == -1) {
fprintf(stderr, Pid %d Exited?\n, pid);
return 0;
}
return sched_class != SCHED_OTHER;
}
struct rt_process_info
{
/* This flag is set for all found processes at the beginning of the
* process list update. Processes that do not have this flag set will
* be assumed dead and removed from the list. The flag is cleared after
* each list update. */
int alive;
int centStamp;
pid_t pid;
pid_t ppid;
gid_t gid;
unsigned int userTime;
unsigned int sysTime;
unsigned int vmSize; // enough?
unsigned int vmRss; // enough?
float sysLoad;
float userLoad;
float cpu_usage;
};
#define MAX_RT_PROCESSES 200
struct rt_process_info rt_process[MAX_RT_PROCESSES]; /* pid alive == 0 */
struct rt_process_info *find_process(pid_t pid)
{
unsigned ix;
for (ix = 0; ix MAX_RT_PROCESSES; ix++)
{
if (rt_process[ix].pid == pid) {
rt_process[ix].alive = 1;
return rt_process[ix];
}
}
return NULL;
}
struct rt_process_info *new_process(pid_t pid)
{
unsigned ix;
for (ix = 0; ix MAX_RT_PROCESSES; ix++)
{
if (rt_process[ix].pid == 0) {
rt_process[ix].pid = pid
Re: [linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
I noticed that I was not on the list... But I have now read the archives. Kernel changes are not workable - at least not short testm! It will take ages until that change is available on most computers. (But if it is the only solution then it is...) Local DOS/security exploits has traditionally been seen as equally dangerous in Unix land - computers often run on Universities... This is the major reason for Linux to be secure even remote. (if you can not trust the security for a user logged how can you trust the security of the whole system if the web server is run as a normal user?) And remember this IS perceived as serious enough for the developers of KDE to disable the RT possibility for arts! The only thing the sound servers need to be able to handle RT audio are: * one of the RT scheduling methods. * locked memory. (Not protected at the moment - hard limits?) This can be archived with a wrapper that drops ALL other priorities before running the real application. - If that drop is done in a safe manner, we have come a long way. At that point the server with its plugins, can do two things a normal process can not: * can dry out the CPU * can lock up memory - force the system into an out of memory situation * can fool the monitor into doing something stupid I have tried to handle this issues in my monitor: * runs on higher priority than the RT audio application * only keeps info about RT processes - ignores others. * uses only static memory structures - no need to allocate unavailable memory... - harder to fool into allocating huge amounts of memory itself... * reduces RT priority of all RT processes at once - harder to hide the attacking process Things missing: * should use a random checking interval. * should also monitor the locked memory (RSS) of RT processes. * it should not reduce the priority of system RT processes since that might be the reason for the DOS attack. [contradicion!] * it should have a log for the system administrator to look in. /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] [RFC] On the issue of only loading trusted code (was: RT monitor)
Stefan wrote: 2. if you try to audit all code that gets loaded and executed with RT prio, and only load trusted code, you're facing an endless task This might actually be an option... If you let the _server_ examine the code on load / execution. Since it is possible to intercept external calls - like fopen(). Recent messages show that we even can let the server compile the plugin - runtime / compiletime processing proof of concept And since it does the compilation it can chose which headers are allowed. /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] [SOURCE] rt monitor to kill runaway RT processes
Hi,
One problem with audio systems that
a) runs as RT processes
b) allows user loaded plugins
Is that it is very simple to make a DOS (Denial of service) attack.
This has resulted in that the feature to run arts in RT mode has been
disabled... see artswrapper.c, search for NO_MORE_LOCAL_DOS_HOLE
I made this code as a proof of concept for a monitor that catches and
reduces priority for processes that misuse this feature.
(this version reduces the priority for all RT processes at once...)
I have been away for some time, so I do not know what happened to it.
But I see that the RT possibility is still removed from arts (Stefan?)
But I figured out that those of you that develops RT plugins might find
it useful even in this form.
Note:
* it has not been tested on SMP (it is likely to be buggy...)
* if your runaway RT program runs on highest SCHED_FIFO priority
this monitor can not help you.
/RogerL
--
Roger Larsson
Skellefteå
Sweden
/* RT monitor.
Copyright (c) 2002 Roger Larsson [EMAIL PROTECTED]
This program is free software; you can redistribute it and/or
modify it under the terms of version 2 of the GNU General Public
License as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
Thanks to autor of KSysGuard Chris Schlaeger for borrowed code...
*/
#include sys/types.h
#include sched.h
#include stdio.h
#include dirent.h
#include sys/mman.h
int isRT(pid_t pid)
{
int sched_class = sched_getscheduler( pid);
if (sched_class == -1) {
fprintf(stderr, Pid %d Exited?\n, pid);
return 0;
}
return sched_class != SCHED_OTHER;
}
struct rt_process_info
{
/* This flag is set for all found processes at the beginning of the
* process list update. Processes that do not have this flag set will
* be assumed dead and removed from the list. The flag is cleared after
* each list update. */
int alive;
int centStamp;
pid_t pid;
pid_t ppid;
gid_t gid;
unsigned int userTime;
unsigned int sysTime;
unsigned int vmSize; // enough?
unsigned int vmRss; // enough?
float sysLoad;
float userLoad;
float cpu_usage;
};
#define MAX_RT_PROCESSES 200
struct rt_process_info rt_process[MAX_RT_PROCESSES]; /* pid alive == 0 */
struct rt_process_info *find_process(pid_t pid)
{
unsigned ix;
for (ix = 0; ix MAX_RT_PROCESSES; ix++)
{
if (rt_process[ix].pid == pid) {
rt_process[ix].alive = 1;
return rt_process[ix];
}
}
return NULL;
}
struct rt_process_info *new_process(pid_t pid)
{
unsigned ix;
for (ix = 0; ix MAX_RT_PROCESSES; ix++)
{
if (rt_process[ix].pid == 0) {
rt_process[ix].pid = pid;
rt_process[ix].alive = 2;
return rt_process[ix];
}
}
return NULL;
}
float cpu_usage(struct rt_process_info *ps)
{
#define BUFSIZE 1024
char buf[BUFSIZE];
FILE *fd;
char status;
unsigned int userTime, sysTime;
snprintf(buf, BUFSIZE - 1, /proc/%d/stat, ps-pid);
buf[BUFSIZE - 1] = '\0';
if ((fd = fopen(buf, r)) == 0)
return (-1);
if (fscanf(fd, %*d %*s %c %d %d %*d %*d %*d %*u %*u %*u %*u %*u %d %d
%*d %*d %*d %*d %*u %*u %*d %u %u,
status, (int*) ps-ppid, (int*) ps-gid,
userTime, sysTime, ps-vmSize,
ps-vmRss) != 7) {
fclose(fd);
return (-1);
}
if (fclose(fd))
return (-1);
{
unsigned int newCentStamp;
int timeDiff, userDiff, sysDiff;
struct timeval tv;
gettimeofday(tv, 0);
newCentStamp = tv.tv_sec * 100 + tv.tv_usec / 1;
// calculate load
if (ps-alive == 2)
ps-sysLoad = ps-userLoad = 0.0f; /* can't give relieable number at the moment... */
else {
timeDiff = (int)(newCentStamp - ps-centStamp);
userDiff = userTime - ps-userTime;
sysDiff = sysTime - ps-sysTime;
if ((timeDiff 0) (userDiff = 0) (sysDiff = 0)) /* protect from bad data */
{
ps-userLoad = ((double) userDiff / timeDiff) * 100.0;
ps-sysLoad = ((double) sysDiff / timeDiff) * 100.0;
}
else
ps-sysLoad = ps-userLoad = 0.0;
}
// update fields
ps-centStamp = newCentStamp;
ps-userTime = userTime;
ps-sysTime = sysTime;
}
ps-cpu_usage = ps-userLoad + ps-sysLoad;
return ps-cpu_usage;
}
/* process reading code from ksysguard */
float cpu_rt_usage(struct rt_process_info **rt_list_head)
{
// Watch out for SMP effects...
float result = 0.0f;
pid_t myself = getpid();
DIR* dir;
struct dirent* entry;
/* read in current process list via the /proc filesystem entry */
if ((dir
Re: [linux-audio-dev] request to use latencytest (OSDL)
On Tuesday 09 July 2002 04.10, [EMAIL PROTECTED] wrote: I've been a grateful user of latencytest for audio setup. I've just started working for the Open Source Development Lab (www.osdl.org) which is a non-profit supporting kernel development by providing hardware and a Scalable Test Platform (www.osdl.org/stp) I'm working on adding tests to the STP, and latencytest looks like it would be very useful, since we don't have many other scheduler focused tests (yet). You might make it even more useful by using it twice... One time with RT prio - testing latency for RT processes And once without - testing latency for ordinary processes Make sure to minimize the idle burning of CPU in both cases - it is not interesting and make the concurrently run testcase look bad... I'd like to include it in the platform. At a minimum this would mean putting a copy in our CVS ( cvs.stp.sourceforge.net:/cvsroot/stp ) and tweaking up a set of automated scripts for the test engine. I'm not sure of all the etiquette when borrowing/seizing GPL code for another GPL project, so i would appreciate help/advice. Asking is nice but not strictly necessary... I've written Takashi and benno, but no response yet. I would have to receive the blessing and permission of the authors before doing anything with the code - are there any other authors, or concerned citizens? My guess is that they will be happy to see it in more regular use... An alternative would be Andrew Mortons amlat - it is simpler, possibly better in an automated environment. Also, more general: Is this code really useful from the kernel perspective? In other words, is there a reason _not to use this test? Latency monitoring is good to do during all the other test cases... since a change that gets you better throughput might destroy interactive behavior... But those tests are probably better done by adding some code in kernel. (The simplest way that I have used is to start a timer when the need_reshed flag is set, and stop it when schedule is called... /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] SuSE 8.0
On Sunday 14 April 2002 22.47, Jörn Nettingsmeier wrote: all in all, a nice distro, if a little bloated. if you're a pssionate do-it-yourselfer, i'd recommend against it, but if you want a quick start and can tolerate excessive eye-candy, suse80 might be for you. I am a do it myself - sometimes... When I want to... When I do not feel like it I like to have as much as possible precompiled on one DVD... /RogerL
Re: [linux-audio-dev] FREENIX paper about scheduling latency and stuff
Nice report, nice tool :-) But not that much data from Windows 2000... (when things get tough - using disk or network, there are no data on Windows 2000 - why? Is it too good or too bad?) /RogerL PS Nice to see that you have not ignored Benno Senoners latencytest DS On måndagen den 8 april 2002 22.49, John Regehr wrote: Hi all, I'm audio-ignorant so I mostly lurk on this list, but I do usually follow discussions about scheduling latency and stuff like that. I thought some members of this list would be interested in a paper I'm working on that's going to appear in the FREENIX track at the USENIX Technical Conference in June. It's about Hourglass, a user-level tool for measuring scheduling behavior. There are some comparisons in the paper between some of the different real-time enhanced versions of Linux. There's a draft here: http://www.cs.utah.edu/~regehr/hourglass-040802.ps http://www.cs.utah.edu/~regehr/hourglass-040802.pdf The final copy is due in about a week; I thought I'd post a copy now in case people find any mistakes, since this will give me the opportunity to correct them. NOTE: This draft will disappear a few days after this message is posted. Anyone reading this message in the archives after April 16 2002 should instead look for the final version of the paper at the Hourglass site: http://www.cs.utah.edu/~regehr/hourglass Thanks, John Regehr Postdoc, School of Computing, University of Utah -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Fwd: [PATCH] Preemptive Kernel for Ingo's O(1) scheduler
Hi low latency lovers, With the improved throughput in dbench I think this got a better chance to be included ... :-) /RogerL -- Forwarded Message -- Subject: [PATCH] Preemptive Kernel for Ingo's O(1) scheduler Date: 11 Jan 2002 16:04:54 -0500 From: Robert Love [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] A version of preempt-kernel is now available for Ingo's O(1) scheduler. For 2.5.2-pre11: ftp://ftp.kernel.org/pub/linux/kernel/people/rml/preempt-kernel/v2.5/preempt- kernel-rml-2.5.2-pre11-1.patch For 2.4.18-pre3 + sched-O1-H6: ftp://ftp.kernel.org/pub/linux/kernel/people/rml/preempt-kernel/v2.4/ingo-O1- sched/preempt-kernel-rml-2.4.18-pre3-ingo-1.patch Because of changes in load_balance, I suggest not using preempt-kernel and an additional sched update on 2.5. I'll update as the patches are merged into 2.5. Making the kernel fully preemptible should be synergistic with Ingo's scheduler with its faster task dispatch time and better RT support. Getting the two to play together was not hard, albeit a bit of a pain. The actually scheduling support is less, due to the simplified schedule and schedule_tail, although there is added code for making the per-CPU runqueues preempt-safe. Benchmarks: 2.5.2-pre11 dbench 16: 24.5364 MB/s 2.5.2-pre11-preempt dbench 16: 27.5192 MB/s 2.5.2-pre11 latencytest: worst-case latency is 18.7ms with 96% scheduling latency on-time 2.5.2-pre11-preempt latencytest: 6ms (1.5ms in all but disk write) with 99.9% scheduling latency on-time 2.5.2-pre11-preempt avg latency is 1.1ms (for an arbitray work-load I tested with). The obstacle for sub-ms average latency is still the long-held spinlocks that can be 100ms+. Full ChangeLog: - make preempt-kernel and Ingo's O(1) scheduler play nicely - (2.5 only) more include additions - various cleanups and such Comments, patches, etc. are appreciated. While it is running stable here in both SMP and UP on both 2.4 and 2.5, more testing could reveal problems. Also, some optimization could be done at this point to hopefully reduce overhead. Enjoy, Robert Love - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ --- -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Kernel developments: disk scheduler (Jens Axboe)
it wants this request to be
@@ -1701,7 +1713,7 @@
*/
queue_nr_requests = 64;
if (total_ram MB(32))
- queue_nr_requests = 256;
+ queue_nr_requests = 512;
/*
* Batch frees according to queue length
diff -ur -X exclude /opt/kernel/linux-2.5.2-pre7/include/linux/elevator.h
linux/include/linux/elevator.h ---
/opt/kernel/linux-2.5.2-pre7/include/linux/elevator.h Fri Jan 4 02:31:31
2002 +++ linux/include/linux/elevator.hFri Jan 4 03:11:13 2002
@@ -60,6 +60,49 @@
#define elv_linus_sequence(rq) ((long)(rq)-elevator_private)
/*
+ * multi-list I/O scheduler
+ */
+extern elevator_t elevator_jens;
+
+struct elv_jens_entry {
+ struct request *rq;
+ unsigned long expire_time;
+ struct list_head fifo_list;
+};
+
+struct elv_jens_data {
+ /*
+* next request to be queued
+*/
+ struct list_head *next_entry;
+
+ /*
+* currently restarted from fifo head due to starvation
+*/
+ int fifo_count;
+
+ /*
+* time based fifo queue, for read and write. could be one list
+* or a better data structure, since it should be just sorted
+* by expire time. later :-)
+*/
+ struct list_head fifo_r;
+ struct list_head fifo_w;
+
+ /*
+* available entries
+*/
+ struct list_head free_entry;
+
+ /*
+* I/O scheduler settings
+*/
+ unsigned long expires[2];
+ int batch;
+ int restart : 1;
+};
+
+/*
* use the /proc/iosched interface, all the below is history -
*/
typedef struct blkelv_ioctl_arg_s {
--
Jens Axboe
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
---
--
Roger Larsson
Skellefteå
Sweden
[linux-audio-dev] Kernel developments: process scheduler (Ingo Molnar)
a series of tests): vanilla-2.5.2-pre6: ./lat_proc fork Process fork+exit: 440. microseconds ./lat_proc exec Process fork+execve: 491.6364 microseconds ./lat_proc shell Process fork+/bin/sh -c: 3545. microseconds O(1)-schedule-2.5.2-pre6: ./lat_proc fork Process fork+exit: 168.6667 microseconds ./lat_proc exec Process fork+execve: 279.6500 microseconds ./lat_proc shell Process fork+/bin/sh -c: 2874. microseconds the difference is pretty dramatic - it's mostly due to avoiding much of the COW overhead that comes from fork()+execve(). The fork()+exit() improvement is mostly due to better CPU affinity - parent and child are running on the same CPU, while the old scheduler pushes the child to another, idle CPU, which creates heavy interlocking traffic between the MM structures. the compilation benchmarks i ran gave very similar results for both schedulers. The O(1) scheduler has a small 2% advantage in make -j benchmarks (not accounting statistical noise - it's hard to produce reliable compilation benchmarks) - probably due to better SMP affinity again. Status == i've tested the new scheduler under the aforementioned range of systems and workloads, but it's still experimental code nevertheless. I've developed it on SMP systems using the 2.5.2-pre kernels, so it has the most testing there, but i did a fair number of UP and 2.4.17 tests as well. NOTE! For the 2.5.2-pre6 kernel to be usable you should apply Andries' latest 2.5.2pre6-kdev_t patch available at: http://www.kernel.org/pub/linux/kernel/people/aeb/ i also tested the RT scheduler for various situations such as sched_yield()-ing of RT tasks, strace-ing RT tasks and other details, and it's all working as expected. There might be some rough edges though. Comments, bug reports, suggestions are welcome, Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ --- -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Re: Scheduler ( was: Just a second ) ...
This might be of interest on linux-audio-dev too... On Tuesday den 18 December 2001 07.09, Linus Torvalds wrote: On Mon, 17 Dec 2001, William Lee Irwin III wrote: 5: 46490271 XT-PIC soundblaster Approximately 4 times more often than the timer interrupt. That's not nice... Yeah. Well, looking at the issue, the problem is probably not just in the sb driver: the soundblaster driver shares the output buffer code with a number of other drivers (there's some horrible dmabuf.c code in common). And yes, the dmabuf code will wake up the writer on every single DMA complete interrupt. Considering that you seem to have them at least 400 times a second (and probably more, unless you've literally had sound going since the machine was booted), I think we know why your setup spends time in the scheduler. On Mon, Dec 17, 2001 at 08:27:18PM -0800, Linus Torvalds wrote: Which sound driver are you using, just in case this _is_ the reason? SoundBlaster 16 A change of hardware should help verify this. A number of sound drivers will use the same logic. You may be able to change this more easily some other way, by using a larger fragment size for example. That's up to the sw that actually feeds the sound stream, so it might be your decoder that selects a small fragment size. Quite frankly I don't know the sound infrastructure well enough to make any more intelligent suggestions about other decoders or similar to try, at this point I just start blathering. But yes, I bet you'll also see much less impact of this if you were to switch to more modern hardware. grep grep grep.. Oh, before you do that, how about changing min_fragment in sb_audio.c from 5 to something bigger like 9 or 10? That audio_devs[devc-dev]-min_fragment = 5; literally means that your minimum fragment size seems to be a rather pathetic 32 bytes (which doesn't mean that your sound will be set to that, but it _might_ be). That sounds totally ridiculous, but maybe I've misunderstood the code. I think it really is 32 samples, yes that is little - but too small? It depends on the used sample frequency... Paul Davis wrote this on linux-audio-dev 2001-12-05 in doing lots of testing on JACK, i've noticed that although the trident driver now works (there were some patches from jaroslav and myself), in general i still get xruns with the lowest possible latency setting for that card (1.3msec per interrupt, 2.6msec buffer). with the same settings on my hammerfall, i don't get xruns, even with substantial system load. Jeff, you've worked on the sb code at some point - does it really do 32-byte sound fragments? Why? That sounds truly insane if I really parsed that code correctly. That's thousands of separate DMA transfers and interrupts per second.. Lets see: we have 1 GHz CPU and interrupts at 1000 Hz = 1 Mcycle / interrupt - is that insane? If the hardware can support it? Why not let it? It is really up to the applications/user to decide... Raising that min_fragment thing from 5 to 10 would make the minimum DMA buffer go from 32 bytes to 1kB, which is a _lot_ more reasonable (what, at 2*2 bytes per sample and 44kHz would mean that a 1kB DMA buffer empties in less than 1/100th of a second, but at least it should be 200 irqs/sec rather than 400). Yes, it is probably more reasonable - but if the soundcard can support it? (I have a vision of lots of linux-audio-dev folks pulling out their new soundcard and replacing it with their since long forgotten SB16...) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Re: [Alsa-devel] RFC: Preemption on interrupt in Linux kernel
On Wednesdayen den 12 December 2001 11.19, you wrote: At Tue, 11 Dec 2001 17:34:34 -0500, Paul Davis wrote: I'm thinking for the benefits that low latency audio applications will have from preemption of running task in favour of the waiting task just at end of interrupt handler execution (instead of end of time slice). I don't believe that such a mechanism is already in place in linux kernel but I may be wrong. i think that there is such code in the LL patches. the return to user space codepath does a conditional_schedule() there. i think. if not, it was a feature of ingo's original LL patch series. As far as looking at the code, Andrew's patch doesn't include this feature. It was only in Ingo's. Takashi Hmm... This should already be working for the cases when the running task is running in user space. See code in file linux/arch/i386/kernel/entry.S ret_from_intr jumps to ret_from_sys_call if the current process executes in user space. In ret_from_sys_call the need_resched flag is checked... The need_resched flag is set if a process with higher priority (goodness) is woken up last line in function reshedule_idle linux/kernel/schedule.c. [called from try_to_wake_up, called from wake_up, called when a process was waiting for something that is now ready] BUT this does not work if the running process was running in the kernel. But note that the need_resched flag is checked when leaving. This is where the patches are comes in: * Ingo Molnars - inserting checks for need_resched wherever it was needed in the kernel. * Andrew Morton - inserting in the most frequently used places only. But note the - Don't do this list * Montavista/Robert Love - allow reschedules in kernel, by using the SMP locks as a preemtion not allowed indicator. Additional context switch locks needed where per CPU data was used. And some checks for need_resched are probably needed in some long held places - like VM handling, walking the list of all memory... Another thing: note that you have no guarantees when not running as RT process. And finally: there is NO need to use the HIGHEST SCHED_FIFO priority only because the latencytest program does. Aim for middle. Increase it slightly if your execution time is really short. Decrease if long. Why? Because your process is most likely the only RT process in the system and will be selected before any non RT process. But if there is another RT process, you need to cooperate somewhat. -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] Compiling LL kernel for AMD without 3DNOW
On Wednesdayen den 14 November 2001 11.36, you wrote: Hello, In reply to D. Stimits and Roger Larsson about compiling an LL kernel for AMD Athlon without the 3DNOW optimizations I still get the undefined references to the *_mmx_* functions. I am perfectly sure that I really recompiled it all, and to demonstrate it, here follow the steps I toke: - - - So I am really wondering now: what am I doing different? I mean you _did_ indeed compile the kernel for AMD without 3DNOW, did you? Maarten I have not tried to compile it since I do not have an Athlon... My guess is: * that there is a inline routine in a header that uses the mmx stuff - but without the check for the 3DNOW flag or * that unsetting the 3DNOW flag removes ordinary mmx functions too... (since you know what mmx functions it complains about you can grep -r mmx-fun linux to find out where they are defined / used / ...) /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] terrible latencytest results.. why??
Some other code has it compiled in. Recompile it all. make clean make ... (I do not know when make dep is required - this might be one situation...) /RogerL On Tuesday 13 November 2001 16:41, you wrote: I now try to compile my kernel for AMD K7, but without 3DNow disabled. Check if you have this in .config CONFIG_X86_USE_3DNOW=y I remove this line, and get the following undefined references: ld -m elf_i386 -T /mnt/hda8/src/linux-2.4.14-LL/arch/i386/vmlinux.lds -e stext arch/i386/kernel/head.o arch/i386/kernel/init_task.o init/main.o init/version.o \ --start-group \ arch/i386/kernel/kernel.o arch/i386/mm/mm.o kernel/kernel.o mm/mm.o fs/fs.o ipc/ipc.o \ drivers/char/char.o drivers/block/block.o drivers/misc/misc.o drivers/net/net.o drivers/media/media.o drivers/char/agp/agp.o drivers/char/drm/drm.o drivers/ide/idedriver.o drivers/cdrom/driver.o drivers/pci/driver.o drivers/video/video.o \ net/network.o \ /mnt/hda8/src/linux-2.4.14-LL/arch/i386/lib/lib.a /mnt/hda8/src/linux-2.4.14-LL/lib/lib.a /mnt/hda8/src/linux-2.4.14-LL/arch/i386/lib/lib.a \ --end-group \ -o vmlinux arch/i386/kernel/kernel.o: In function `machine_real_restart': arch/i386/kernel/kernel.o(.text+0x102): undefined reference to `_mmx_memcpy' arch/i386/kernel/kernel.o(.text+0x165): undefined reference to `_mmx_memcpy' arch/i386/kernel/kernel.o: In function `copy_segments': arch/i386/kernel/kernel.o(.text+0x488): undefined reference to `_mmx_memcpy' arch/i386/kernel/kernel.o: In function `copy_thread': arch/i386/kernel/kernel.o(.text+0x534): undefined reference to `_mmx_memcpy' arch/i386/kernel/kernel.o: In function `dump_extended_fpu': arch/i386/kernel/kernel.o(.text+0x6f82): undefined reference to `_mmx_memcpy' arch/i386/kernel/kernel.o(.text.init+0xab1): more undefined references to `_mmx_memcpy' follow arch/i386/kernel/kernel.o(__ksymtab+0x170): undefined reference to `mmx_clear_page' arch/i386/kernel/kernel.o(__ksymtab+0x178): undefined reference to `mmx_copy_page' kernel/kernel.o: In function `mm_init': kernel/kernel.o(.text+0x165f): undefined reference to `_mmx_memcpy' kernel/kernel.o: In function `copy_files': kernel/kernel.o(.text+0x1c64): undefined reference to `_mmx_memcpy' kernel/kernel.o(.text+0x1ca4): undefined reference to `_mmx_memcpy' kernel/kernel.o: In function `do_fork': kernel/kernel.o(.text+0x215b): undefined reference to `_mmx_memcpy' kernel/kernel.o: In function `sys_create_module': kernel/kernel.o(.text+0x3588): undefined reference to `_mmx_memcpy' kernel/kernel.o(.text+0x4a07): more undefined references to `_mmx_memcpy' followmm/mm.o: In function `do_wp_page': mm/mm.o(.text+0xe88): undefined reference to `mmx_clear_page' mm/mm.o(.text+0xe9a): undefined reference to `mmx_copy_page' mm/mm.o: In function `do_anonymous_page': mm/mm.o(.text+0x124b): undefined reference to `mmx_clear_page' mm/mm.o: In function `do_no_page': mm/mm.o(.text+0x1339): undefined reference to `mmx_copy_page' mm/mm.o: In function `pte_alloc': mm/mm.o(.text+0x14a0): undefined reference to `mmx_clear_page' mm/mm.o: In function `get_zeroed_page': mm/mm.o(.text+0x9a41): undefined reference to `mmx_clear_page' mm/mm.o: In function `shmem_getpage_locked': mm/mm.o(.text+0xc28b): undefined reference to `mmx_clear_page' mm/mm.o: In function `shmem_symlink': mm/mm.o(.text+0xcdc4): undefined reference to `_mmx_memcpy' mm/mm.o(.text+0xce79): undefined reference to `_mmx_memcpy' fs/fs.o: In function `block_symlink': fs/fs.o(.text+0x49b4): undefined reference to `_mmx_memcpy' fs/fs.o: In function `flush_old_exec': fs/fs.o(.text+0x834a): undefined reference to `_mmx_memcpy' fs/fs.o: In function `d_alloc': fs/fs.o(.text+0x11818): undefined reference to `_mmx_memcpy' fs/fs.o(.text+0x11d48): more undefined references to `_mmx_memcpy' follow make: *** [vmlinux] Error 1 -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] terrible latencytest results.. why??
On Friday 09 November 2001 09:35, Maarten de Boer wrote: Roger Larsson [EMAIL PROTECTED] wrote: They are not bad either... (sub 10 ms) I think it is bad. I want the 3 ms :-) I will be added latency in the processing (fft, ifft) and it all adds up. Something is definitely very wrong here. I am running kernel 2.4.13 patched with preempt-kernel-rml-2.4.13-2.patch And you have enabled it in .config? (checking the obvious...) Yeah, I triple checked the obvious (also: did I run lilo, did lilo use the right vmlinuz) But wait... an AMD They have some 3DNow specific kernel optimizations that causes trouble - they need to disable preemtion (uses fp registers...) Hmm.. Does anybody have more information about this? Could I disable these optimizations compiling for another CPU (386, or maybe even Pentium Classic?) I suppose the performance penalty that would give would be acceptable. Latency is more important. Check if you have this in .config CONFIG_X86_USE_3DNOW=y Would Andrew Morton's LL patches also be affected by this? Yes, if this is the problem. /RogerL -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] latency statistics
On Sunday 28 October 2001 09:16, dave willis wrote: due to popular demand (and not-so-popular criticism), i have done the please forgive me... :-) latency tests properly, using both alsa and oss. you can see them from my computer here: http://66.68.113.152/latency-results/ I can not reach the results - Timeout on server i'm only posting this to linux-audio-dev, linux-audio-user, and alsa-devel, so feel free to forward it elsewhere, or put it on your linux audio site (and save my bandwidth). i used several different kernels and patches, as indicated on the main page. fwiw, dave -- Roger Larsson Skellefteå Sweden
Re: [linux-audio-dev] alsa latency statistics
On Wednesday 24 October 2001 03:54, dave willis wrote: - - - latencytest was ran as follows: ./latencytest -t alsa -d hw -q none 3 256 ./latencytest -t oss -q none 3 256 Question is, what did you do while this did run? * Compile the kernel? * run x11perf? There are lots of testcases included in the latencytest package. The point with the patches are to give a better worst case! The normal case when nothing else is being run will not change. (latency as a term is difficult. The audio latency in this case is close to buffer latency= 4.000 ms - i.e. from generation to actual sending to soundcard, then add some time for output filters...) - - - used alsa-cvs from oct 20, oss = alsa's oss-emu Tests with standard oss would be interesting too... /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] Fwd: Updated Linux 2.4.9/2.4.10 kernel preemption patches
Hi all, A little note from the linux-kernel list. See http://kpreempt.sourceforge.net/ for description. This is the low-latency work that is most likely to bee included in linux-2.5 (Note the spikes some part of the code is holding a spin-lock for too long... But the nice part is that - to get good SMP performance they will have to go!) /RogerL -- Forwarded Message -- Subject: Updated Linux 2.4.9/2.4.10 kernel preemption patches Date: 29 Aug 2001 01:35:26 -0400 From: Robert Love [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Updated patches are at: http://tech9.net/rml/linux/patch-rml-2.4.9-ac3-preempt-kernel-1 and, http://tech9.net/rml/linux/patch-rml-2.4.10-pre2-preempt-kernel-1 for kernels 2.4.9-ac3 and 2.4.10-pre2. These are updates of Nigel Gamble's kernel preemption patches for recent kernels. See http://kpreempt.sourceforge.net/. These patches create a configure option to enable a preemptible kernel using SMP lock points. A preemptible kernel will yield control of execution to higher priority processes as needed. Ie, the process timeslice now applies to kernel space. Changes since my previous patch: * update for 2.4.9-ac3 and 2.4.10-pre2 * fix the compile bug (yay!) -- the linking dependency of dec_and_lock requires CONFIG_HAVE_DEC_LOCK which SMP sets in recent kernels. now CONFIG_PREEMPT sets, too So, yes, this should fix the kernel compile buggy. At least it did for me, after I was finally able to reproduce the problem. Enjoy and please comment, test, and benchmark. -- Robert M. Love rml at ufl.edu rml at tech9.net - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ --- -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] linux-kernel info: 2.4 Performance bug in read ahead
Hi, I have for some time wondered why reading two big (and identical) files simultaneously (diff) takes the double time of reading them sequentially... today I found the cause :-) Linux limits the read ahead to 1+31 pages (á 4096 bytes = 128 kB) with modern HD this is read quickly (4.3 ms, with 30 MB/s) then the head has to move to the other file... (and that might take longer!) You can hear the head moving when running diff on two big files... (or running ardour from 24... :-) Do you get around half the throughput when reading from more than one file? I sent a patch to linux-kernel that corrects this (I changed the limit to 512 pages = 2MB). But this can give a latency problem... S streams reading full amount S*2MB will takes S*60 ms. But the kernel will adopt a suitable read ahead for each sequential stream... More about this later (including a patch) when the jury has returned... /RogerL -- Roger Larsson Skellefteå Sweden
[linux-audio-dev] linux-kernel info: 2.4 Performance bug in read ahead
Hi, I have for some time wondered why reading two big (and identical) files takes the double time of comparing them... today I found the cause :-) Linux limits the read ahead to 1+31 pages (á 4096 bytes = 128 kB) with modern HD this is read quickly (4.3 ms, with 30 MB/s) then the head has to move... You can hear the head moving when running diff on two big files... I sent a patch to linux-kernel that corrects this (I changed the limit to 512 pages = 2MB). But this can give a latency problem... S streams reading full amount S*2MB will takes S*60 ms. But the kernel will adopt a suitable read ahead for each sequential stream... More about this later (including a patch) when the jury has returned... /RogerL -- Roger Larsson Skellefteå Sweden
[with RFC] Developers - Priorities... Was: Re: [linux-audio-dev] user lowlatency kernel experience
Do both csound and timidity run with MAX SCHED_FIFO prio? Then this will be a big problem... I guess most people has followed Bennos example code to the exact word - including running max prio. * Developers - you should not do this... (It is fine when testing for the absolute best possible latencies...) Most systems do not have any processes running SCHED_FIFO so using the minimum SCHED_FIFO priority should be enough. Let the highest priorities be free for stuff were it really matters - like a process for monitoring audio processes and killing hung ones... (if the hung one has max prio no monitor can get a time slice to be able to determine that it is hung...) General rule: shorter bursts gives you the right to use higher prios long calculation burst requires low prios - can still be SCHED_FIFO. but, more important stuff get the right to use higher prios... - seriously, audio stuff are not that important... Here comes the RFCs part: I have been playing with the thought of implementing a virtual SCHED class SCHED_MM were the priority requested really has to be backed up by how the processor is used - a priority matches a max burst time and a min sleep. If you try to use the processor for longer or more often that that your process will be killed! (some sort of burst CPU quota) Since it will have those strict rules it should be possible to use from a program without root privileges... (requests can be rejected if the total requested CPU usage is over some configurable limit) BUT - how to configure the time in such a way that a process can request a priority and know what it will get, note that processors might change their frequency nowadays... /RogerL On Wednesday 13 June 2001 15:30, you wrote: hello, would just like to tell you of my experience with a low-latency kernel. i tried kernel 245 with the low-latency patch. to test the system i tried using timidity and csound at the same time, driven by jazz. both csound and timidity run as root with the high priority scheduling activated. my experience was that it became very easy to hang the system.if csound got to many notes, it would render forever end never return, thus effectively hanging my machine. since the increase in performance wasnt really all that great in my case, i have now disabled the low-latency flag, fortunately this can be done without recompiling the kernel. so, since people here seem to think that the low-latency kernel is really great, i suspect i have missed something. is there a way to get processes to get sceduled promptly, but at the same time not hog the cpu if they try?
