On Wed, Feb 13, 2013 at 10:26 AM, Michael Haberler <[email protected]>wrote:
> > We have a report from 'the field' which we cannot make sense of. > > The situation: > - an AMD board: http://www.asus.com/Motherboard/F1A75M_PRO > - dmesg post boot: http://pastebin.com/38XrxNBy > - xeno-regression-test runs well, max 32us jitter > - John's Xenomai kernel packages: 3.5.7/2.6.2.1 [1] > - a native-skin userland RT threads application (linuxcnc[3]) > - 2 threads > - jitter measured with its own GUI application 'latency-test' > - successfully tested on several other platforms > > > what we observed: > > 1. Problem behaviour > --------------------- > - boot > - run LinuxCNC latency-test > - observe massive spikes in latency > - >100uS on a 25uS thread! > - http://static.mah.priv.at/public/latency/skunkworks-unprimed.png > > now any of 2), 3) or 4) improve latency: > > 2. run switchtest: temporary change > ------------------------------------ > - while still running LinuxCNC latency-test from 1) above, > - running "/usr/lib/xenomai/testsuite/switchtest -s 1000" in a separate > window > - hit 'Reset Statistics' on the latency-test window > - max latency drops massively > - see http://static.mah.priv.at/public/latency/skunkworks-primed.png [3] > - ^C-ing out of the switchtest makes latency rise again > > > 3. running a trivial shell script: temporary change during script > execution > > ---------------------------------------------------------------------------- > - reboot > - run latency-test, again observe latency spikes > - in a separate window, run: > - while true; do echo "nothing" > /dev/null; done > - again, latency-test shows rather low latency figures after hitting > 'reset statistics' *as long as the above script is running* > - quote from Sam: "BTW - I ran the latency-test all night with the > donothing scrip and it peaked at about 19.6us latency." > - killing the script makes the latency spikes reappear. > > > 4. running xeno-regression-test and breaking out: permanent drop in latency > -------------------------------------------------------------------------- > - reboot > - run latency-test, again observe latency spikes > - in a separate window, run: > sudo xeno-regression-test -l "/usr/lib/xenomai/testsuite/dohell -m /tmp > 100 " -t 2 > - latency drops > - the key observation: if you break by ^C out of xeno-regression-test, > *latency > figures remain low* > - note that breaking out of xeno-regression-test left some processes > running, obviously dd and ls: http://pastebin.ca/2313116 > - once these processes complete ( http://pastebin.ca/2313117) latency > goes up again. > > second data point: > we have a report from another user, same kernel, Intel Q8200 Quad core > board, which confirms 'dohell 900' in a separate window does drop latency > significantly. This suggests it might not board specific. > > > This leaves us puzzled as to the causality here. We would really like to > get rid of the latency spikes, but the shell script approach isnt appealing. > > Any suggestions? > I've seen similar behaviour. In my case it had to do with the latency of transitions of the cpu's idle states. The problem was worked around by providing "nohlt idle=poll". I'm sure it is documented somewhere on the xenomai website too. Hope this helps, Henri. > > - Michael, Sam, John > > ---- > > [1] Config: > > http://www.zultron.com/static/2013/01/xenomai/3.5.7-test-32-bit/config-3.5.7.txt > PPA info: http://wiki.linuxcnc.org/cgi-bin/wiki.pl?XenomaiKernelPackages > > [2] this screenshot was taken before we isolated the improvement to > running switchtest -s 1000; before we had used 'sudo > xeno-regression-test -l "/usr/lib/xenomai/testsuite/dohell -m /tmp 100" > -t 2' and it was observed that breaking out of the test with ^C towards > its end improves latency. > > [3] the logic in essence follows trivial-periodic.c. While I have not > separated out the test into a simple program, this is certainly doable > if required. > > > > > > > _______________________________________________ > Xenomai mailing list > [email protected] > http://www.xenomai.org/mailman/listinfo/xenomai > _______________________________________________ Xenomai mailing list [email protected] http://www.xenomai.org/mailman/listinfo/xenomai
