Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Le vendredi 12 octobre 2007 à 23:58 +0200, Philippe Gerum a écrit : On Fri, 2007-10-12 at 23:51 +0200, Stelian Pop wrote: On Fri, Oct 12, 2007 at 10:22:45PM +0200, Stelian Pop wrote: Or even go crazy and activate SMP again ? Would have been too easy: # modprobe xeno_native Xenomai: native skin init failed, code -19. The I-pipe likely told Xenomai that the LAPIC was unusable, because it has been put in dummy state by the clock event layer. Must be something silly going on at I-pipe level. Please switch HPET off just for kicks. No, it doesn't change a thing. What does change something however is CONFIG_X86_UP_IOAPIC. Without it (UP or UP+APIC), latency works (albeit with higher latencies - up to 30 us - than with 2.6.20+xeno-2.3.4 - where I saw latencies up to 20-25 _in_SMP_mode_). With IOAPIC I get code 19. -- Stelian Pop [EMAIL PROTECTED] ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Le vendredi 12 octobre 2007 à 22:22 +0200, Stelian Pop a écrit : I still have a strange problem though: after loading xeno_nucleus and before loading xeno_native, the keyboard reacts strangely: each key press results in at least 6 (and up to 20) letters on the terminal. FYI, I am no longer able to reproduce this keyboard problem today. Could be because or a temporary USB problem, or because I did a cold boot this morning... Or it could be anything else... -- Stelian Pop [EMAIL PROTECTED] ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Stelian Pop wrote: Le vendredi 12 octobre 2007 à 23:58 +0200, Philippe Gerum a écrit : On Fri, 2007-10-12 at 23:51 +0200, Stelian Pop wrote: On Fri, Oct 12, 2007 at 10:22:45PM +0200, Stelian Pop wrote: Or even go crazy and activate SMP again ? Would have been too easy: # modprobe xeno_native Xenomai: native skin init failed, code -19. The I-pipe likely told Xenomai that the LAPIC was unusable, because it has been put in dummy state by the clock event layer. Must be something silly going on at I-pipe level. Please switch HPET off just for kicks. No, it doesn't change a thing. What does change something however is CONFIG_X86_UP_IOAPIC. Without it (UP or UP+APIC), latency works (albeit with higher latencies - up to 30 us - than with 2.6.20+xeno-2.3.4 - where I saw latencies up to 20-25 Those 5 us are likely due to the programming overhead of the PIT (in contrast to the fast IO-APIC in LAPIC/SMP setups). _in_SMP_mode_). With IOAPIC I get code 19. Where do you get ENODEV? On nucleus startup? Please provide /proc/timer_list output of the working and non-working setups. Jan PS: For unknown reasons your mails don't make it to my web.de address, only to the list. Do you get any error messages? ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Sat, Oct 13, 2007 at 04:42:27PM +0200, Jan Kiszka wrote: Where do you get ENODEV? On nucleus startup? Please provide /proc/timer_list output of the working and non-working setups. It turns out that I had the Linux NMI watchdog enabled (nmi_watchdog=1 on the command line) and this was causing the -ENODEV problems. Once removed, I'm able to boot and successfully run all configurations: UP, UP + APIC, UP + APIC + IO_APIC, SMP. And the latencies are back to normal. Maybe we should detect that the NMI watchdog is enabled and issue a warning message, this would save others a few hours and many kernel builds... This is with your timer cleanup patch, of course. PS: For unknown reasons your mails don't make it to my web.de address, only to the list. Do you get any error messages? I did get one saying: [EMAIL PROTECTED]: host mx-ha02.web.de[217.72.192.188] refused to talk to me: 554 Transaction failed. For explanation visit http://freemail.web.de/reject/?ip=88.191.70.230 I didn't investigate yet what's happenning. -- Stelian Pop [EMAIL PROTECTED] ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Sat, 2007-10-13 at 18:12 +0200, Stelian Pop wrote: On Sat, Oct 13, 2007 at 04:42:27PM +0200, Jan Kiszka wrote: Where do you get ENODEV? On nucleus startup? Please provide /proc/timer_list output of the working and non-working setups. It turns out that I had the Linux NMI watchdog enabled (nmi_watchdog=1 on the command line) and this was causing the -ENODEV problems. Once removed, I'm able to boot and successfully run all configurations: UP, UP + APIC, UP + APIC + IO_APIC, SMP. And the latencies are back to normal. The clock event layer turns off the APIC timer when the NMI watchdog is set to use the IO-APIC. Unfortunately, using nmi_watchdog=2 may just not work at all on broken hardware with recent kernels (like the one I have just in front of me...). Maybe we should detect that the NMI watchdog is enabled and issue a warning message, this would save others a few hours and many kernel builds... There is work in progress on the NMI issue, but this message will likely appear until we've fixed this properly. This is with your timer cleanup patch, of course. PS: For unknown reasons your mails don't make it to my web.de address, only to the list. Do you get any error messages? I did get one saying: [EMAIL PROTECTED]: host mx-ha02.web.de[217.72.192.188] refused to talk to me: 554 Transaction failed. For explanation visit http://freemail.web.de/reject/?ip=88.191.70.230 I didn't investigate yet what's happenning. -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Stelian Pop wrote: On Sat, Oct 13, 2007 at 04:42:27PM +0200, Jan Kiszka wrote: Where do you get ENODEV? On nucleus startup? Please provide /proc/timer_list output of the working and non-working setups. It turns out that I had the Linux NMI watchdog enabled (nmi_watchdog=1 on the command line) and this was causing the -ENODEV problems. Once removed, I'm able to boot and successfully run all configurations: UP, UP + APIC, UP + APIC + IO_APIC, SMP. And the latencies are back to normal. Maybe we should detect that the NMI watchdog is enabled and issue a warning message, this would save others a few hours and many kernel builds... NMI is still broken for = 2.6.22, known issue :-/. The problem is that Linux falls back to the PIT, leaving our beloved APIC in disabled state which makes Xenomai fail when asking for it. $Someone has to look into the reason why we cannot take over the APIC in that scenario... This is with your timer cleanup patch, of course. PS: For unknown reasons your mails don't make it to my web.de address, only to the list. Do you get any error messages? I did get one saying: [EMAIL PROTECTED]: host mx-ha02.web.de[217.72.192.188] refused to talk to me: 554 Transaction failed. For explanation visit http://freemail.web.de/reject/?ip=88.191.70.230 I didn't investigate yet what's happenning. Your IP is probably associated to some DSL access, and a lot of providers block such senders categorically due to all the spam robots running on hijacked user PCs. Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Hi Jan, [taking this on the list after several mails with Philippe...] Le jeudi 11 octobre 2007 à 22:47 +0200, Jan Kiszka a écrit : This patch for SVN trunk fixes most of the current bugs around hardware timer takeover and release from/to Linux. [...] I have a problem with the timer on my MacBook Pro (Core2Duo, used in _32_ bit mode)(*): when Xenomai takes over the timer (at 'modprobe xeno_native' time), the Linux timer stops. Looking into /proc/xenomai/irq shows that Xenomai does receive the hardware interrupts, and /proc/interrupts shows that they are no longer forwarded to Linux. Before loading xeno_native, everything is ok. Linux userspace continues to somewhat work: I can issue commands, and depending on the syscalls they made I suppose (no, strace doesn't work), sometimes they end correctly sometimes they hang (and I cannot interrupt them by ^C or other signals.). I tried several .config variations, without any change in behaviour: my current test config has SMP, NO_HZ, APIC, PREEMPT, HIRES all disabled). This happens with a 2.6.22.9 kernel, adeos-ipipe-2.6.22-i386-1.10-07, xenomai svn HEAD (rev 3050), with or without your current patch. It is quite possible that this is not a new problem, since I have this laptop since a few weeks only and I never ran Xenomai on it. I'll happily provide any further information or test results if you need. Thanks, Stelian. (*) yeah, I know I could install a x86_64 distribution but I had some terrible experiences in the past - mainly due to the usage of some proprietary bricks like flash, java etc. I know some of those have been resolved today, and one of these days I should try once more, but now it is not a good time for this. -- Stelian Pop [EMAIL PROTECTED] ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Philippe Gerum wrote: On Thu, 2007-10-11 at 22:47 +0200, Jan Kiszka wrote: This patch for SVN trunk fixes most of the current bugs around hardware timer takeover and release from/to Linux. Tested and found working here (including SMP): - 2.6.22, APIC, highres=off, nohz=off - 2.6.22, APIC, highres=on, nohz=on - 2.6.20, APIC Tests to be done: - 2.6.22, PIT (currently building...) - 2.6.20, PIT Things became quite complex in i386/hal.c now. Some of this complexity might be avoidable if RTHAL_APIC_TIMER_VECTOR equalled LOCAL_TIMER_VECTOR. What's the reason for this? Is it something related to pre-highres times of Linux (ie. 2.6.20 and earlier)? Can we overcome it, at least for recent kernels? The way it works resembles the way Linux works around an issue raised by broken APIC hardware, whose timer bluntly stalls when entering C3. For this reason, Linux keeps the i8253 as the master tick device, and broadcasts the APIC-based local timer vector on all CPUs from the tick handler. Not doing so would stop the timekeeping when entering a sleep mode (this what the tick-broadcast mode is about with generic clock events support in recent kernels). This said, we do rely on the APIC timer to program the delivery of RTHAL_APIC_TIMER interrupts in oneshot mode, so the above work around does not help us a lot when it comes to C3 on broken hardware anyway. (Not to speak of the TSC which may stop when entering C2 or get corrupted in C3 in many cases too...) Another issue to take into account is the cost of timekeeping through a Xenomai host timer and explicit propagation of a faked LOCAL_TIMER interrupt via the I-pipe, from the real-time domain to the root domain (what we would have to do in order to recycle the local timer vector), compared to the cost of letting the Linux timekeeping stuff live its own life in parallel, without any intervention from the Xenomai side. To sum up, I'd say that we could work the way we are already running in PIT mode, and relay host ticks to Linux, freeing the local timer interrupt for our own use. But this may also be more expensive for the Sorry, I can't follow your argumentation at this point: For 2.6.22, we are now already relaying the Linux ticks, for _all_ configurations. In the rare case that Linux decides to use the PIT (e.g. because the NMI watchdog is active), the APIC does not work for us right now anyway. So, if we are already relaying the host timer, my question remains why we need to use a different APIC timer vector in this case. If we may decide to enhance I-pipe in a way that it redirect Linux to the PIT clockevent driver in case we want to use the APIC, than this is a different discussion, of course. real-time side. We are lacking some benchmarks here, but this could be tested quite easily (we would need to disable IRQ0 when the host ticking service is handed over to Xenomai though). Yeah, good question: What is cheaper latency-wise, timer separation in hardware or stacking in software? Performance-wise I think the switch-back to the PIT is not optimal for the overall system. Jan -- Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Stelian Pop wrote: Hi Jan, [taking this on the list after several mails with Philippe...] Le jeudi 11 octobre 2007 à 22:47 +0200, Jan Kiszka a écrit : This patch for SVN trunk fixes most of the current bugs around hardware timer takeover and release from/to Linux. [...] I have a problem with the timer on my MacBook Pro (Core2Duo, used in _32_ bit mode)(*): when Xenomai takes over the timer (at 'modprobe xeno_native' time), the Linux timer stops. Hmm, that's not too different from my own test setup. Looking into /proc/xenomai/irq shows that Xenomai does receive the hardware interrupts, and /proc/interrupts shows that they are no longer forwarded to Linux. Before loading xeno_native, everything is ok. Linux userspace continues to somewhat work: I can issue commands, and depending on the syscalls they made I suppose (no, strace doesn't work), sometimes they end correctly sometimes they hang (and I cannot interrupt them by ^C or other signals.). I tried several .config variations, without any change in behaviour: my current test config has SMP, NO_HZ, APIC, PREEMPT, HIRES all disabled). This happens with a 2.6.22.9 kernel, adeos-ipipe-2.6.22-i386-1.10-07, xenomai svn HEAD (rev 3050), with or without your current patch. It is quite possible that this is not a new problem, since I have this laptop since a few weeks only and I never ran Xenomai on it. I'll happily provide any further information or test results if you need. /proc/xenomai/timerstat/master may provide further hints about the state of the host timer (please keep my patch applied for this). Also, you could take an I-pipe trace around the timer takeover and the following few milliseconds, using the new trigger feature: echo rthal_timer_request /proc/ipipe/trace/trigger BTW, does the latency test of Xenomai work? Jan -- Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Philippe, Jan Kiszka wrote: This patch for SVN trunk fixes most of the current bugs around hardware timer takeover and release from/to Linux. Tested and found working here (including SMP): - 2.6.22, APIC, highres=off, nohz=off - 2.6.22, APIC, highres=on, nohz=on - 2.6.20, APIC Tests to be done: - 2.6.22, PIT (currently building...) - 2.6.20, PIT To close this topic: Also the tests with APIC switched off were successfully completed yesterday. So the logic should be fine now, ... Things became quite complex in i386/hal.c now. Some of this complexity might be avoidable if RTHAL_APIC_TIMER_VECTOR equalled LOCAL_TIMER_VECTOR. What's the reason for this? Is it something related to pre-highres times of Linux (ie. 2.6.20 and earlier)? Can we overcome it, at least for recent kernels? So, while this patch may work correctly, my feeling is it needs yet another round of polishing, also looking for now possibly outdated comments. ...just this aspect needs to be addressed. Jan -- Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Thu, 2007-10-11 at 22:47 +0200, Jan Kiszka wrote: This patch for SVN trunk fixes most of the current bugs around hardware timer takeover and release from/to Linux. Tested and found working here (including SMP): - 2.6.22, APIC, highres=off, nohz=off - 2.6.22, APIC, highres=on, nohz=on - 2.6.20, APIC Tests to be done: - 2.6.22, PIT (currently building...) - 2.6.20, PIT Things became quite complex in i386/hal.c now. Some of this complexity might be avoidable if RTHAL_APIC_TIMER_VECTOR equalled LOCAL_TIMER_VECTOR. What's the reason for this? Is it something related to pre-highres times of Linux (ie. 2.6.20 and earlier)? Can we overcome it, at least for recent kernels? The way it works resembles the way Linux works around an issue raised by broken APIC hardware, whose timer bluntly stalls when entering C3. For this reason, Linux keeps the i8253 as the master tick device, and broadcasts the APIC-based local timer vector on all CPUs from the tick handler. Not doing so would stop the timekeeping when entering a sleep mode (this what the tick-broadcast mode is about with generic clock events support in recent kernels). This said, we do rely on the APIC timer to program the delivery of RTHAL_APIC_TIMER interrupts in oneshot mode, so the above work around does not help us a lot when it comes to C3 on broken hardware anyway. (Not to speak of the TSC which may stop when entering C2 or get corrupted in C3 in many cases too...) Another issue to take into account is the cost of timekeeping through a Xenomai host timer and explicit propagation of a faked LOCAL_TIMER interrupt via the I-pipe, from the real-time domain to the root domain (what we would have to do in order to recycle the local timer vector), compared to the cost of letting the Linux timekeeping stuff live its own life in parallel, without any intervention from the Xenomai side. To sum up, I'd say that we could work the way we are already running in PIT mode, and relay host ticks to Linux, freeing the local timer interrupt for our own use. But this may also be more expensive for the real-time side. We are lacking some benchmarks here, but this could be tested quite easily (we would need to disable IRQ0 when the host ticking service is handed over to Xenomai though). -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Fri, 2007-10-12 at 11:47 +0200, Jan Kiszka wrote: Philippe Gerum wrote: On Thu, 2007-10-11 at 22:47 +0200, Jan Kiszka wrote: This patch for SVN trunk fixes most of the current bugs around hardware timer takeover and release from/to Linux. Tested and found working here (including SMP): - 2.6.22, APIC, highres=off, nohz=off - 2.6.22, APIC, highres=on, nohz=on - 2.6.20, APIC Tests to be done: - 2.6.22, PIT (currently building...) - 2.6.20, PIT Things became quite complex in i386/hal.c now. Some of this complexity might be avoidable if RTHAL_APIC_TIMER_VECTOR equalled LOCAL_TIMER_VECTOR. What's the reason for this? Is it something related to pre-highres times of Linux (ie. 2.6.20 and earlier)? Can we overcome it, at least for recent kernels? The way it works resembles the way Linux works around an issue raised by broken APIC hardware, whose timer bluntly stalls when entering C3. For this reason, Linux keeps the i8253 as the master tick device, and broadcasts the APIC-based local timer vector on all CPUs from the tick handler. Not doing so would stop the timekeeping when entering a sleep mode (this what the tick-broadcast mode is about with generic clock events support in recent kernels). This said, we do rely on the APIC timer to program the delivery of RTHAL_APIC_TIMER interrupts in oneshot mode, so the above work around does not help us a lot when it comes to C3 on broken hardware anyway. (Not to speak of the TSC which may stop when entering C2 or get corrupted in C3 in many cases too...) Another issue to take into account is the cost of timekeeping through a Xenomai host timer and explicit propagation of a faked LOCAL_TIMER interrupt via the I-pipe, from the real-time domain to the root domain (what we would have to do in order to recycle the local timer vector), compared to the cost of letting the Linux timekeeping stuff live its own life in parallel, without any intervention from the Xenomai side. To sum up, I'd say that we could work the way we are already running in PIT mode, and relay host ticks to Linux, freeing the local timer interrupt for our own use. But this may also be more expensive for the Sorry, I can't follow your argumentation at this point: For 2.6.22, we are now already relaying the Linux ticks, for _all_ configurations. In the rare case that Linux decides to use the PIT (e.g. because the NMI watchdog is active), the APIC does not work for us right now anyway. I will happily see this legacy code improved if we can do this without adding too much complexity. The recent surge of bugs in this area pleads for both sides. So, yes, we do relay ticks when generic clock events are available, this was a recent change of mine when porting over this infrastructure for 2.6.22. But as you know, this won't work for anything earlier, until we move the whole damn thing under host tick emulation for everyone. And to get to that point, we will first need to recycle the local timer for every purpose, when the APIC is enabled, as you initially suggested. Making this change is a no-brainer implementation-wise, I'm just -reasonably- worried about the performance cost imposed on the real-time side. Hence the open question I raised. So, if we are already relaying the host timer, my question remains why we need to use a different APIC timer vector in this case. If we may decide to enhance I-pipe in a way that it redirect Linux to the PIT clockevent driver in case we want to use the APIC, than this is a different discussion, of course. real-time side. We are lacking some benchmarks here, but this could be tested quite easily (we would need to disable IRQ0 when the host ticking service is handed over to Xenomai though). Yeah, good question: What is cheaper latency-wise, timer separation in hardware or stacking in software? Performance-wise I think the switch-back to the PIT is not optimal for the overall system. Except that in such a case, we don't need any non-preemptible real-time code to be executed in order to schedule the interrupt for processing by the kernel. Faking the interrupt through propagation to each CPU would clearly add more pressure on the nklock too, even if we try hard to stagger the per-CPU clocks to avoid obvious contentions. IOW, moving everything on top of Xenomai's core timer would cause Xenomai to spend some time in a non-preemptible way in order to relay the tick to Linux for each CPU, and that could have an impact latency-wise, maybe. We need benchmark figures to sort out this issue, at least to measure the cost induced by host tick emulation on top of the Xenomai domain, including in SMP mode. Jan -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Fri, 2007-10-12 at 15:19 +0200, Stelian Pop wrote:
On Fri, Oct 12, 2007 at 01:15:31PM +0200, Jan Kiszka wrote:
[EMAIL PROTECTED] cat /proc/ipipe/trace/max
/frozen holds the result. Also, a bit more post_trace_points may help to
see what goes on after that function is called.
Ah, sorry. Here it comes, with post_trace_points set to 100:
[EMAIL PROTECTED]:~# echo 1 /proc/ipipe/trace/enable
[EMAIL PROTECTED]:~# echo 100 /proc/ipipe/trace/post_trace_points
[EMAIL PROTECTED]:~# echo rthal_timer_request /proc/ipipe/trace/trigger
[EMAIL PROTECTED]:~# modprobe xeno_native
[EMAIL PROTECTED]:~# cat /proc/ipipe/trace/frozen
I-pipe frozen back-tracing service on 2.6.22.9-xeno/ipipe-1.10-07
It looks like we are still using the TSC emulation over the 8254; I
don't ever recall if it's supposed to work actually. Now that the
remaining known issues have been fixed by Jan's latest patch, could you
switch your CPU config to CORE2, instead of 486? Maybe we'll get lucky.
CPU: 0, Freeze: 533994599352 cycles, Trace Points: 100 (+100)
Calibrated minimum trace-point overhead: 0.094 us
+- Hard IRQs ('|': locked)
|+ unused
||+--- unused
|||+-- Xenomai
+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
|+-- Delay flag ('+': 1 us, '!': 10
us)
||+- NMI noise ('N')
|||
TypeUser Val. TimeDelay Function (Parent)
:| #begin 0x8000 -370.094 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -370.184 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -370.089 ipipe_check_context+0xc
(cache_alloc_refill+0x406)
:#func -370.094 ipipe_check_context+0xc
(cache_alloc_refill+0x418)
:#func -370.094 ipipe_check_context+0xc
(cache_alloc_refill+0x441)
:#func -370.139 ipipe_check_context+0xc
(cache_alloc_refill+0x66)
:#func -360.109 ipipe_check_context+0xc
(cache_alloc_refill+0x165)
:#func -360.099 __ipipe_restore_root+0x8
(__kmalloc+0x90)
:#func -360.099 __ipipe_unstall_root+0x8
(__ipipe_restore_root+0x27)
:| #begin 0x8000 -360.094 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -360.209 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -360.384 xnheap_init+0xe (xnpod_init+0x243)
:+func -35+ 7.850 init_extent+0xe (xnheap_init+0x111)
:+func -280.144 xnthread_init+0xe (xnpod_init+0x2af)
:+func -270.119 __xntimer_init+0xe
(xnthread_init+0x53)
:+func -270.084 snprintf+0xb (__xntimer_init+0x96)
:+func -270.159 vsnprintf+0xe (snprintf+0x22)
:+func -270.259 number+0xe (vsnprintf+0x2d8)
:| +begin 0x8000 -270.129 __xntimer_init+0x14b
(xnthread_init+0x53)
:| *+func -270.134 __ipipe_restore_pipeline_head+0xd
(__xntimer_init+0x117)
:| +end 0x8000 -270.159 __ipipe_restore_pipeline_head+0x9e
(__xntimer_init+0x117)
:+func -260.094 __xntimer_init+0xe
(xnthread_init+0x9b)
:+func -260.074 snprintf+0xb (__xntimer_init+0x96)
:+func -260.104 vsnprintf+0xe (snprintf+0x22)
:+func -260.219 number+0xe (vsnprintf+0x2d8)
:| +begin 0x8000 -260.129 __xntimer_init+0x14b
(xnthread_init+0x9b)
:| *+func -260.084 __ipipe_restore_pipeline_head+0xd
(__xntimer_init+0x117)
:| +end 0x8000 -260.443 __ipipe_restore_pipeline_head+0x9e
(__xntimer_init+0x117)
:+func -250.234 xnregistry_init+0xb
(xnpod_init+0x359)
:+func -250.094 rthal_apc_alloc+0xe
(xnregistry_init+0x1c)
:| +begin 0x8000 -250.129 rthal_apc_alloc+0xbb
(xnregistry_init+0x1c)
:| *+func -250.124 __ipipe_restore_pipeline_head+0xd
(rthal_apc_alloc+0xa7)
:| +end 0x8000 -250.239 __ipipe_restore_pipeline_head+0x9e
(rthal_apc_alloc+0xa7)
:+func -240.229 create_proc_entry+0xd
(xnregistry_init+0x3e)
:+func -240.219 proc_create+0xe
(create_proc_entry+0x54)
:+func -240.099 __kmalloc+0xe (proc_create+0x8f)
:+func -240.089 ipipe_check_context+0xc
(__kmalloc+0xb4)
:+func -240.104 ipipe_check_context+0xc
(__kmalloc+0x5c)
:#func -240.074 __ipipe_restore_root+0x8
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Stelian Pop wrote: On Fri, Oct 12, 2007 at 01:15:31PM +0200, Jan Kiszka wrote: [EMAIL PROTECTED] cat /proc/ipipe/trace/max /frozen holds the result. Also, a bit more post_trace_points may help to see what goes on after that function is called. Ah, sorry. Here it comes, with post_trace_points set to 100: ... :+func -60.209 xnarch_get_cpu_time+0x8 (xnpod_enable_timesource+0xa0) :+func -60.094 rthal_get_8254_tsc+0xe (xnarch_get_cpu_time+0xd) Ah, CONFIG_M486. Try to pick a less generic CPU type, maybe TSC-less support got broken recently. Jan -- Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
Stelian Pop wrote: On Fri, Oct 12, 2007 at 11:14:43AM +0200, Jan Kiszka wrote: Hmm, that's not too different from my own test setup. I'm attaching the .config and dmesg in case you see something strange. /proc/xenomai/timerstat/master may provide further hints about the state of the host timer (please keep my patch applied for this). /proc/xenomai/timerstat/master says: CPU SCHEDULED FIRED TIMEOUTINTERVAL HANDLER NAME 01 1 - - NULL [host-timer] Also, you could take an I-pipe trace around the timer takeover and the following few milliseconds, using the new trigger feature: echo rthal_timer_request /proc/ipipe/trace/trigger It gives this: [EMAIL PROTECTED] cat /proc/ipipe/trace/max /frozen holds the result. Also, a bit more post_trace_points may help to see what goes on after that function is called. BTW, does the latency test of Xenomai work? No. It hangs after warming up. I'm able to interrupt with ^C and then it prints a single line showing a max latency of 208983.492 ms (same value on several invocations). Ah, then we may fail to program the APIC appropriately. That would need a closer look if you want to dig into this. /me is going to be distracted from this for now. Jan -- Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Fri, Oct 12, 2007 at 03:41:19PM +0200, Philippe Gerum wrote:
It looks like we are still using the TSC emulation over the 8254; I
don't ever recall if it's supposed to work actually. Now that the
remaining known issues have been fixed by Jan's latest patch, could you
switch your CPU config to CORE2, instead of 486? Maybe we'll get lucky.
You won't get away so easy. CPU switched back to CORE2, same behaviour.
New trace:
[EMAIL PROTECTED]:~# cat /proc/ipipe/trace/frozen
I-pipe frozen back-tracing service on 2.6.22.9-xeno/ipipe-1.10-07
CPU: 0, Freeze: 224810852352 cycles, Trace Points: 100 (+100)
Calibrated minimum trace-point overhead: 0.090 us
+- Hard IRQs ('|': locked)
|+ unused
||+--- unused
|||+-- Xenomai
+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
|+-- Delay flag ('+': 1 us, '!': 10 us)
||+- NMI noise ('N')
|||
TypeUser Val. TimeDelay Function (Parent)
:+func -320.090 ipipe_check_context+0xc
(kmem_cache_alloc+0x27)
:#func -320.070 __ipipe_restore_root+0x8
(kmem_cache_alloc+0x68)
:#func -310.085 __ipipe_unstall_root+0x8
(__ipipe_restore_root+0x27)
:| #begin 0x8000 -310.085 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -310.170 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -310.090 ipipe_check_context+0xc
(cache_alloc_refill+0x48e)
:#func -310.100 ipipe_check_context+0xc
(cache_alloc_refill+0x4a0)
:#func -310.095 ipipe_check_context+0xc
(cache_alloc_refill+0x4c9)
:#func -310.120 ipipe_check_context+0xc
(cache_alloc_refill+0x6b)
:#func -310.140 ipipe_check_context+0xc
(cache_alloc_refill+0x162)
:#func -310.080 __ipipe_restore_root+0x8
(__kmalloc+0x8a)
:#func -310.085 __ipipe_unstall_root+0x8
(__ipipe_restore_root+0x27)
:| #begin 0x8000 -300.085 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -300.220 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -300.390 xnheap_init+0x14 (xnpod_init+0x24b)
:+func -30+ 7.836 init_extent+0xe (xnheap_init+0x118)
:+func -220.320 xnthread_init+0xe (xnpod_init+0x2b5)
:+func -220.145 __xntimer_init+0xe (xnthread_init+0x58)
:+func -210.090 snprintf+0xb (__xntimer_init+0x96)
:+func -210.120 vsnprintf+0xe (snprintf+0x22)
:+func -210.220 number+0xe (vsnprintf+0x2f9)
:| +begin 0x8000 -210.115 __xntimer_init+0x14b
(xnthread_init+0x58)
:| *+func -210.130 __ipipe_restore_pipeline_head+0x11
(__xntimer_init+0x119)
:| +end 0x8000 -210.175 __ipipe_restore_pipeline_head+0x9e
(__xntimer_init+0x119)
:+func -210.110 __xntimer_init+0xe (xnthread_init+0x9f)
:+func -200.075 snprintf+0xb (__xntimer_init+0x96)
:+func -200.140 vsnprintf+0xe (snprintf+0x22)
:+func -200.240 number+0xe (vsnprintf+0x2f9)
:| +begin 0x8000 -200.125 __xntimer_init+0x14b
(xnthread_init+0x9f)
:| *+func -200.085 __ipipe_restore_pipeline_head+0x11
(__xntimer_init+0x119)
:| +end 0x8000 -200.475 __ipipe_restore_pipeline_head+0x9e
(__xntimer_init+0x119)
:+func -190.205 xnregistry_init+0xb (xnpod_init+0x360)
:+func -190.095 rthal_apc_alloc+0x14
(xnregistry_init+0x1c)
:| +begin 0x8000 -190.175 rthal_apc_alloc+0xbc
(xnregistry_init+0x1c)
:| *+func -190.120 __ipipe_restore_pipeline_head+0x11
(rthal_apc_alloc+0xaf)
:| +end 0x8000 -190.125 __ipipe_restore_pipeline_head+0x9e
(rthal_apc_alloc+0xaf)
:+func -190.205 create_proc_entry+0xd
(xnregistry_init+0x3e)
:+func -180.215 proc_create+0x14
(create_proc_entry+0x43)
:+func -180.095 __kmalloc+0xe (proc_create+0x95)
:+func -180.090 ipipe_check_context+0xc
(__kmalloc+0xb2)
:+func -180.080 ipipe_check_context+0xc
(__kmalloc+0x4e)
:#func -180.080 __ipipe_restore_root+0x8
(__kmalloc+0x8a)
:#func -180.095 __ipipe_unstall_root+0x8
(__ipipe_restore_root+0x27)
:| #begin 0x8000 -180.090 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -180.155
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Fri, Oct 12, 2007 at 01:15:31PM +0200, Jan Kiszka wrote:
[EMAIL PROTECTED] cat /proc/ipipe/trace/max
/frozen holds the result. Also, a bit more post_trace_points may help to
see what goes on after that function is called.
Ah, sorry. Here it comes, with post_trace_points set to 100:
[EMAIL PROTECTED]:~# echo 1 /proc/ipipe/trace/enable
[EMAIL PROTECTED]:~# echo 100 /proc/ipipe/trace/post_trace_points
[EMAIL PROTECTED]:~# echo rthal_timer_request /proc/ipipe/trace/trigger
[EMAIL PROTECTED]:~# modprobe xeno_native
[EMAIL PROTECTED]:~# cat /proc/ipipe/trace/frozen
I-pipe frozen back-tracing service on 2.6.22.9-xeno/ipipe-1.10-07
CPU: 0, Freeze: 533994599352 cycles, Trace Points: 100 (+100)
Calibrated minimum trace-point overhead: 0.094 us
+- Hard IRQs ('|': locked)
|+ unused
||+--- unused
|||+-- Xenomai
+- Linux ('*': domain stalled, '+': current, '#': current+stalled)
|+-- Delay flag ('+': 1 us, '!': 10 us)
||+- NMI noise ('N')
|||
TypeUser Val. TimeDelay Function (Parent)
:| #begin 0x8000 -370.094 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -370.184 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -370.089 ipipe_check_context+0xc
(cache_alloc_refill+0x406)
:#func -370.094 ipipe_check_context+0xc
(cache_alloc_refill+0x418)
:#func -370.094 ipipe_check_context+0xc
(cache_alloc_refill+0x441)
:#func -370.139 ipipe_check_context+0xc
(cache_alloc_refill+0x66)
:#func -360.109 ipipe_check_context+0xc
(cache_alloc_refill+0x165)
:#func -360.099 __ipipe_restore_root+0x8
(__kmalloc+0x90)
:#func -360.099 __ipipe_unstall_root+0x8
(__ipipe_restore_root+0x27)
:| #begin 0x8000 -360.094 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -360.209 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -360.384 xnheap_init+0xe (xnpod_init+0x243)
:+func -35+ 7.850 init_extent+0xe (xnheap_init+0x111)
:+func -280.144 xnthread_init+0xe (xnpod_init+0x2af)
:+func -270.119 __xntimer_init+0xe (xnthread_init+0x53)
:+func -270.084 snprintf+0xb (__xntimer_init+0x96)
:+func -270.159 vsnprintf+0xe (snprintf+0x22)
:+func -270.259 number+0xe (vsnprintf+0x2d8)
:| +begin 0x8000 -270.129 __xntimer_init+0x14b
(xnthread_init+0x53)
:| *+func -270.134 __ipipe_restore_pipeline_head+0xd
(__xntimer_init+0x117)
:| +end 0x8000 -270.159 __ipipe_restore_pipeline_head+0x9e
(__xntimer_init+0x117)
:+func -260.094 __xntimer_init+0xe (xnthread_init+0x9b)
:+func -260.074 snprintf+0xb (__xntimer_init+0x96)
:+func -260.104 vsnprintf+0xe (snprintf+0x22)
:+func -260.219 number+0xe (vsnprintf+0x2d8)
:| +begin 0x8000 -260.129 __xntimer_init+0x14b
(xnthread_init+0x9b)
:| *+func -260.084 __ipipe_restore_pipeline_head+0xd
(__xntimer_init+0x117)
:| +end 0x8000 -260.443 __ipipe_restore_pipeline_head+0x9e
(__xntimer_init+0x117)
:+func -250.234 xnregistry_init+0xb (xnpod_init+0x359)
:+func -250.094 rthal_apc_alloc+0xe
(xnregistry_init+0x1c)
:| +begin 0x8000 -250.129 rthal_apc_alloc+0xbb
(xnregistry_init+0x1c)
:| *+func -250.124 __ipipe_restore_pipeline_head+0xd
(rthal_apc_alloc+0xa7)
:| +end 0x8000 -250.239 __ipipe_restore_pipeline_head+0x9e
(rthal_apc_alloc+0xa7)
:+func -240.229 create_proc_entry+0xd
(xnregistry_init+0x3e)
:+func -240.219 proc_create+0xe
(create_proc_entry+0x54)
:+func -240.099 __kmalloc+0xe (proc_create+0x8f)
:+func -240.089 ipipe_check_context+0xc
(__kmalloc+0xb4)
:+func -240.104 ipipe_check_context+0xc
(__kmalloc+0x5c)
:#func -240.074 __ipipe_restore_root+0x8
(__kmalloc+0x90)
:#func -240.089 __ipipe_unstall_root+0x8
(__ipipe_restore_root+0x27)
:| #begin 0x8000 -240.094 __ipipe_unstall_root+0x4d
(__ipipe_restore_root+0x27)
:| +end 0x8000 -230.144 __ipipe_unstall_root+0x3f
(__ipipe_restore_root+0x27)
:+func -230.209 proc_register+0xe
(create_proc_entry+0x72)
:+func -230.129
Re: [Xenomai-core] [PATCH] fix hw-timer setup/cleanup for i386
On Fri, 2007-10-12 at 23:51 +0200, Stelian Pop wrote: On Fri, Oct 12, 2007 at 10:22:45PM +0200, Stelian Pop wrote: Or even go crazy and activate SMP again ? Would have been too easy: # modprobe xeno_native Xenomai: native skin init failed, code -19. The I-pipe likely told Xenomai that the LAPIC was unusable, because it has been put in dummy state by the clock event layer. Must be something silly going on at I-pipe level. Please switch HPET off just for kicks. This is with all relevant options on (HPET, HIRES, SMP, NO_HZ). I guess I'll have to take one step at a time. -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
