[Xenomai-core] [PATCH] I-pipe hosted tracing service
Hi all, as a late Christmas gift I would like to roll out a probably quite useful instrumentation tool: This is the PREEMPT_RT-inspired I-pipe tracing service! The core ipipe_trace.patch-v4 should apply cleanly against 2.6.14 kernels with ipipe-1.0-12, the instrumentation patches were developed against Xenomai SVN trunk. Currently, this is x86 only, but adding further archs should be quite easy. I can provide two instrumentation experiments so far: ipipe_trace.instr tries to make use of existing statistics capturing points in the i-ipipe patch, but fails to grab the worst case due to spurious path end reports inside the i-pipe core. Anyway, the tracer can at least look a bit before and after a trace (see below) so that you may add the missing delays on a piece of paper on your own (still not optimal, I know). maxlat_proc.patch is actually more than a pure instrumentation based on the NMI watchdog. First it exports the trigger threshold value of the watchdog to a /proc variable which you can now tune during runtime. And then it replaces the rather hard die_nmi() report of deadline misses by an ipipe_trace_freeze() call. After such a hit you can safely look at the reported trace, retune the NMI, and even start a new round. INSTALLATION 1. apply ipipe_trace.patch-v4 against your xeno-prepared RT-kernel 2. switch on CONFIG_IPIPE_TRACE (I-pipe suboption) 3. apply some or all instrumentation patches 4. recompile the kernel API --- The tracer provides 4 instrumentation functions: void ipipe_trace_begin(unsigned long value); Mark the beginning of a critical trace path. All following ipipe_trace_begin() calls are noted but ignored until ipipe_trace_end() is invoked. An optional value can be provided, it will be stored with this trace point. void ipipe_trace_end(unsigned long value); Mark the end of a critical trace path and stores it for dumping in case this is the longest path captured so far. All following ipipe_trace_end() calls are noted but have no further effect until a ipipe_trace_begin() is invoked again. An optional value can be provided, it will be stored with this trace point. void ipipe_trace_freeze(unsigned long value); Freezes a back-trace on invocation. This has higher priority then any ipipe_trace_begin/end marks. Only the first freeze will get stored, all following calls of ipipe_trace_freeze() are noted but ignored until the current frozen path is reset (see below). An optional value can be provided, it will be stored with this trace point. void ipipe_trace_special(unsigned char special_id, unsigned long value); Marks a user-defined trace point by adding the provided information to the trace but does not cause any further effects. USAGE - All tracing related controls and information can be found under /proc/ipipe/trace/. Here is an overview: max - the longest trace path between all ipipe_trace_begin() and ipipe_trace_end() points. Read from it for dumping the content, write any data to it for resetting. frozen - the first frozen back-trace triggered by a ipipe_trace_freeze() call. Read from it for dumping the content, write any data to it for resetting. enable - the global switch to turn all tracing on or off. Default: 1 (on). pre_trace_points - number of additional trace points to be reported before the longest trace path. Default: 10 post_trace_points - number of additional trace points to be captured and reported both for the longest trace path and the frozen path. Default: 10. back_trace_points - number of trace points backwards to be reported for the frozen trace path (including the freeze event itself). Default: 30. verbose - report more details via max or frozen. Just give it a try. Default: 0 (off). Ok, so far for the documentation. I can only recommend to do your own experiments, it can reveal interesting insights into your real-time system! There is indication for hard-IRQ-off phases during a trace and for the rare case the NMI recurses while in a critical tracing phase (mostly, NMI just shows up by its name, i.e. via the related functions). And if you are lucky, you may even see which (Linux) driver was touching the wire when some hardware-related latency bomb goes off :). Also quite useful but not yet ready for release is an export of the ipipe_trace_xxx calls to userspace. Invoking a freeze right after the testsuite's latency tool exceeded some threshold is on my to-do list (just a hack ATM). This may be easily done via that RTDM benchmark device. Here is a tentative trace excerpt of such an instrumentation: TypeTime Function (Parent) : fn -112 __ipipe_stall_root (work_resched) : fn -112! __ipipe_unstall_iret_root (restore_raw) [here we likely left for userspace until the IRQ arrived] :|fn -32 __ipipe_handle_irq (common_interrupt) :|fn -31 __ipipe_ack_system_irq (__ipipe_handle_irq) :|fn
[Xenomai-core] 2.4.32 compilation error + old timer example problem
I checked out xeno.trunk just a few minutes ago. With kernel 2.4.32, kernel compilation fails with this: mq.c: In function `mq_notify': mq.c:475: error: `SIGEV_SIVNAL' undeclared (first use in this function) mq.c:475: error: (Each undeclared identifier is reported only once mq.c:475: error: for each function it appears in.) kernel/xenomai/skins/posix/mq.c SIGEV_SIVNAL = SIGEV_SIGNAL ..with that it succeeds. But still my timer example doesn't work - see: https://mail.gna.org/public/xenomai-help/2005-12/msg00020.html With 2.6.14.3 it works... Thanks for any hints, Happy New Year, Hannes. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
[Xenomai-core] [PATCH] I-pipe hosted tracing service
Hi all, as a late Christmas gift I would like to roll out a probably quite useful instrumentation tool: This is the PREEMPT_RT-inspired I-pipe tracing service! The core ipipe_trace.patch-v4 should apply cleanly against 2.6.14 kernels with ipipe-1.0-12, the instrumentation patches were developed against Xenomai SVN trunk. Currently, this is x86 only, but adding further archs should be quite easy. I can provide two instrumentation experiments so far: ipipe_trace.instr tries to make use of existing statistics capturing points in the i-ipipe patch, but fails to grab the worst case due to spurious path end reports inside the i-pipe core. Anyway, the tracer can at least look a bit before and after a trace (see below) so that you may add the missing delays on a piece of paper on your own (still not optimal, I know). maxlat_proc.patch is actually more than a pure instrumentation based on the NMI watchdog. First it exports the trigger threshold value of the watchdog to a /proc variable which you can now tune during runtime. And then it replaces the rather hard die_nmi() report of deadline misses by an ipipe_trace_freeze() call. After such a hit you can safely look at the reported trace, retune the NMI, and even start a new round. INSTALLATION 1. apply ipipe_trace.patch-v4 against your xeno-prepared RT-kernel 2. switch on CONFIG_IPIPE_TRACE (I-pipe suboption) 3. apply some or all instrumentation patches 4. recompile the kernel API --- The tracer provides 4 instrumentation functions: void ipipe_trace_begin(unsigned long value); Mark the beginning of a critical trace path. All following ipipe_trace_begin() calls are noted but ignored until ipipe_trace_end() is invoked. An optional value can be provided, it will be stored with this trace point. void ipipe_trace_end(unsigned long value); Mark the end of a critical trace path and stores it for dumping in case this is the longest path captured so far. All following ipipe_trace_end() calls are noted but have no further effect until a ipipe_trace_begin() is invoked again. An optional value can be provided, it will be stored with this trace point. void ipipe_trace_freeze(unsigned long value); Freezes a back-trace on invocation. This has higher priority then any ipipe_trace_begin/end marks. Only the first freeze will get stored, all following calls of ipipe_trace_freeze() are noted but ignored until the current frozen path is reset (see below). An optional value can be provided, it will be stored with this trace point. void ipipe_trace_special(unsigned char special_id, unsigned long value); Marks a user-defined trace point by adding the provided information to the trace but does not cause any further effects. USAGE - All tracing related controls and information can be found under /proc/ipipe/trace/. Here is an overview: max - the longest trace path between all ipipe_trace_begin() and ipipe_trace_end() points. Read from it for dumping the content, write any data to it for resetting. frozen - the first frozen back-trace triggered by a ipipe_trace_freeze() call. Read from it for dumping the content, write any data to it for resetting. enable - the global switch to turn all tracing on or off. Default: 1 (on). pre_trace_points - number of additional trace points to be reported before the longest trace path. Default: 10 post_trace_points - number of additional trace points to be captured and reported both for the longest trace path and the frozen path. Default: 10. back_trace_points - number of trace points backwards to be reported for the frozen trace path (including the freeze event itself). Default: 30. verbose - report more details via max or frozen. Just give it a try. Default: 0 (off). Ok, so far for the documentation. I can only recommend to do your own experiments, it can reveal interesting insights into your real-time system! There is indication for hard-IRQ-off phases during a trace and for the rare case the NMI recurses while in a critical tracing phase (mostly, NMI just shows up by its name, i.e. via the related functions). And if you are lucky, you may even see which (Linux) driver was touching the wire when some hardware-related latency bomb goes off :). Also quite useful but not yet ready for release is an export of the ipipe_trace_xxx calls to userspace. Invoking a freeze right after the testsuite's latency tool exceeded some threshold is on my to-do list (just a hack ATM). This may be easily done via that RTDM benchmark device. Here is a tentative trace excerpt of such an instrumentation: TypeTime Function (Parent) : fn -112 __ipipe_stall_root (work_resched) : fn -112! __ipipe_unstall_iret_root (restore_raw) [here we likely left for userspace until the IRQ arrived] :|fn -32 __ipipe_handle_irq (common_interrupt) :|fn -31 __ipipe_ack_system_irq (__ipipe_handle_irq) :|fn
