Re: [Xenomai-core] Fwd: RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK
On Wed, 2007-10-10 at 08:12 +0200, Jan Kiszka wrote: Gilles Chanteperdrix wrote: Jan Kiszka wrote: Again, the priority should not be the issue. The issue is likely that a pending or just being handled non-RT IRQ can stall some RT IRQ at hardware level. That must not happen. I-pipe rather has to log, acknowledge, and possibly mask that line quickly so that RT IRQs can be delivered again. Thinking a bit more about my ethernet vs timer issue. If, when an ethernet interrupt is pending, adeos is not aware that there is also a timer interrupt pending, it will call the ethernet interrupt handler immediately then unmask the interrupt. So, Adeos will never have a chance to handle the timer interrupt before another ethernet interrupt is handled. Ergo, giving the timer interrupt the highest priority is what must be done. No. Adeos will first start to dispatch the Ethernet IRQ. It will ackmask it and then re-enable the IRQ delivery before calling into the handler. Only if the ethernet interrupt is not a real-time event. At this point the hardware can report the timer IRQ, and Adeos will immediately start to deliver that one instead. With IRQ hardware priorities, you only optimise the case when both interrupts are pending in the hardware at the same time. The worst-case remains that the Ethernet IRQ comes first, Adeos starts to handle it, and _then_ the timer IRQ arrives. This is something the hardware can in no way avoid (without looking into the future...). When the processor has a notion of internal priority level which it does inherit from the level of the event it currently processes, the above assumption is wrong. In such a case, the next interrupt to be serviced would be equivalent to pending_IRQ_mask CPU_interrupt_mask processor_level, i.e. multiple high priority interrupts would be processed before a low priority one is eventually triggered. So in that case, Gilles's assertion does make a lot of sense. Jan ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Fwd: RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK
On 10/10/07, Philippe Gerum [EMAIL PROTECTED] wrote: On Wed, 2007-10-10 at 08:12 +0200, Jan Kiszka wrote: Gilles Chanteperdrix wrote: Jan Kiszka wrote: Again, the priority should not be the issue. The issue is likely that a pending or just being handled non-RT IRQ can stall some RT IRQ at hardware level. That must not happen. I-pipe rather has to log, acknowledge, and possibly mask that line quickly so that RT IRQs can be delivered again. Thinking a bit more about my ethernet vs timer issue. If, when an ethernet interrupt is pending, adeos is not aware that there is also a timer interrupt pending, it will call the ethernet interrupt handler immediately then unmask the interrupt. So, Adeos will never have a chance to handle the timer interrupt before another ethernet interrupt is handled. Ergo, giving the timer interrupt the highest priority is what must be done. No. Adeos will first start to dispatch the Ethernet IRQ. It will ackmask it and then re-enable the IRQ delivery before calling into the handler. Only if the ethernet interrupt is not a real-time event. At this point the hardware can report the timer IRQ, and Adeos will immediately start to deliver that one instead. With IRQ hardware priorities, you only optimise the case when both interrupts are pending in the hardware at the same time. The worst-case remains that the Ethernet IRQ comes first, Adeos starts to handle it, and _then_ the timer IRQ arrives. This is something the hardware can in no way avoid (without looking into the future...). When the processor has a notion of internal priority level which it does inherit from the level of the event it currently processes, the above assumption is wrong. In such a case, the next interrupt to be serviced would be equivalent to pending_IRQ_mask CPU_interrupt_mask processor_level, i.e. multiple high priority interrupts would be processed before a low priority one is eventually triggered. So in that case, Gilles's assertion does make a lot of sense. The AT91 interrupts need an EOI, I wonder if we are not simply doing the EOI too late. That is, the EOI should be done when the interrupt has been acked, before it is handled, so that other interrupts can be delivered. -- Gilles Chanteperdrix ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Fwd: RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK
On 10/10/07, Philippe Gerum [EMAIL PROTECTED] wrote: On Wed, 2007-10-10 at 10:54 +0200, Gilles Chanteperdrix wrote: On 10/10/07, Philippe Gerum [EMAIL PROTECTED] wrote: On Wed, 2007-10-10 at 08:12 +0200, Jan Kiszka wrote: Gilles Chanteperdrix wrote: Jan Kiszka wrote: Again, the priority should not be the issue. The issue is likely that a pending or just being handled non-RT IRQ can stall some RT IRQ at hardware level. That must not happen. I-pipe rather has to log, acknowledge, and possibly mask that line quickly so that RT IRQs can be delivered again. Thinking a bit more about my ethernet vs timer issue. If, when an ethernet interrupt is pending, adeos is not aware that there is also a timer interrupt pending, it will call the ethernet interrupt handler immediately then unmask the interrupt. So, Adeos will never have a chance to handle the timer interrupt before another ethernet interrupt is handled. Ergo, giving the timer interrupt the highest priority is what must be done. No. Adeos will first start to dispatch the Ethernet IRQ. It will ackmask it and then re-enable the IRQ delivery before calling into the handler. Only if the ethernet interrupt is not a real-time event. At this point the hardware can report the timer IRQ, and Adeos will immediately start to deliver that one instead. With IRQ hardware priorities, you only optimise the case when both interrupts are pending in the hardware at the same time. The worst-case remains that the Ethernet IRQ comes first, Adeos starts to handle it, and _then_ the timer IRQ arrives. This is something the hardware can in no way avoid (without looking into the future...). When the processor has a notion of internal priority level which it does inherit from the level of the event it currently processes, the above assumption is wrong. In such a case, the next interrupt to be serviced would be equivalent to pending_IRQ_mask CPU_interrupt_mask processor_level, i.e. multiple high priority interrupts would be processed before a low priority one is eventually triggered. So in that case, Gilles's assertion does make a lot of sense. The AT91 interrupts need an EOI, I wonder if we are not simply doing the EOI too late. That is, the EOI should be done when the interrupt has been acked, before it is handled, so that other interrupts can be delivered. The EOI would lower the priority barrier for interrupts, so this makes sense. Looking at the I-pipe code for AT91RM9200 for instance, I only see the AIC being asked to mask the IRQ source as a result of the fast I-pipe acknowledge being called, nothing more. If the AIC needs explicit EOI to lower the priority level which is not covered by the previous action, then it's indeed missing. The latest fix regarding the fasteoi interrupt flow would not have any impact, since AFAICS, we are solely dealing with level interrupts here. The EOI is done by the irq_finish macro, which is called after ipipe_handle_irq. -- Gilles Chanteperdrix ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Fwd: RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK
On Wed, 2007-10-10 at 11:36 +0200, Gilles Chanteperdrix wrote: On 10/10/07, Philippe Gerum [EMAIL PROTECTED] wrote: On Wed, 2007-10-10 at 10:54 +0200, Gilles Chanteperdrix wrote: On 10/10/07, Philippe Gerum [EMAIL PROTECTED] wrote: On Wed, 2007-10-10 at 08:12 +0200, Jan Kiszka wrote: Gilles Chanteperdrix wrote: Jan Kiszka wrote: Again, the priority should not be the issue. The issue is likely that a pending or just being handled non-RT IRQ can stall some RT IRQ at hardware level. That must not happen. I-pipe rather has to log, acknowledge, and possibly mask that line quickly so that RT IRQs can be delivered again. Thinking a bit more about my ethernet vs timer issue. If, when an ethernet interrupt is pending, adeos is not aware that there is also a timer interrupt pending, it will call the ethernet interrupt handler immediately then unmask the interrupt. So, Adeos will never have a chance to handle the timer interrupt before another ethernet interrupt is handled. Ergo, giving the timer interrupt the highest priority is what must be done. No. Adeos will first start to dispatch the Ethernet IRQ. It will ackmask it and then re-enable the IRQ delivery before calling into the handler. Only if the ethernet interrupt is not a real-time event. At this point the hardware can report the timer IRQ, and Adeos will immediately start to deliver that one instead. With IRQ hardware priorities, you only optimise the case when both interrupts are pending in the hardware at the same time. The worst-case remains that the Ethernet IRQ comes first, Adeos starts to handle it, and _then_ the timer IRQ arrives. This is something the hardware can in no way avoid (without looking into the future...). When the processor has a notion of internal priority level which it does inherit from the level of the event it currently processes, the above assumption is wrong. In such a case, the next interrupt to be serviced would be equivalent to pending_IRQ_mask CPU_interrupt_mask processor_level, i.e. multiple high priority interrupts would be processed before a low priority one is eventually triggered. So in that case, Gilles's assertion does make a lot of sense. The AT91 interrupts need an EOI, I wonder if we are not simply doing the EOI too late. That is, the EOI should be done when the interrupt has been acked, before it is handled, so that other interrupts can be delivered. The EOI would lower the priority barrier for interrupts, so this makes sense. Looking at the I-pipe code for AT91RM9200 for instance, I only see the AIC being asked to mask the IRQ source as a result of the fast I-pipe acknowledge being called, nothing more. If the AIC needs explicit EOI to lower the priority level which is not covered by the previous action, then it's indeed missing. The latest fix regarding the fasteoi interrupt flow would not have any impact, since AFAICS, we are solely dealing with level interrupts here. The EOI is done by the irq_finish macro, which is called after ipipe_handle_irq. So you are definitely right. irq_finish must be done before the interrupt is pipelined, right after it has been masked in the fast ack handler. -- Philippe. ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
Re: [Xenomai-core] Fwd: RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK
Gregory CLEMENT wrote: -- Forwarded message -- From: Gregory CLEMENT [EMAIL PROTECTED] Date: 9 oct. 2007 21:07 Subject: Re: [Xenomai-core] RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK To: Jan Kiszka [EMAIL PROTECTED] 2007/10/9, Jan Kiszka [EMAIL PROTECTED]: Gregory CLEMENT wrote: Here, they are the last latency we get on AT91SAM9261-EK. As just now I haven't the board at home, I send the last result we stored. The prority of dbgu should be set to 6 instead of 7, but I can't assure it, for Xenomai. Hmm, hardware interrupt priorities must not impact the worst-case latency if I-pipe acks and mask them appropriately (the worst case is when multiple interrupts happen in a row, not at the same time). But this statement is not based on knowledge about potential pitfalls of this arch. Are there specialities that require this tweaking? Indeed there was little impact on Xenomai, but a big impact in RTAI. I believe that in RTAI (or at least in our RTAI port), there is longer critical section than in Xenomai. We observe big latency when calibrator was writing a lot of \r on serial line, that was causing a lot of interrupts. I think that during a critical section (IT disabled), we get an timer interrupt and a serial interrupt. As serial interrupt has bigger priority it is treated first and as during an interrupt serial driver can send or receive 256 character, it can add a big delay. Again, the priority should not be the issue. The issue is likely that a pending or just being handled non-RT IRQ can stall some RT IRQ at hardware level. That must not happen. I-pipe rather has to log, acknowledge, and possibly mask that line quickly so that RT IRQs can be delivered again. We just revealed a similar problem over i386 about fasteoi-type IRQs. What type of IRQs are involved here? Maybe it's the same bug. first Xenomai: #insmod /lib/modules/2.6.20.13/kernel/drivers/xenomai/testing/xeno_timerbench.ko #cd /usr/xenomai/bin/ Which versions were you using for both tests? Do you still have the involved .configs? Both version are 2.6.20.13 I join the config files. Hmm, I wonder why we have this difference in the configs: # diff -u config-RTAI config-Xenomai [...] @@ -147,7 +243,7 @@ # AT91 Board Options # # CONFIG_MTD_AT91_DATAFLASH_CARD is not set -# CONFIG_MTD_NAND_AT91_BUSWIDTH_16 is not set +CONFIG_MTD_NAND_AT91_BUSWIDTH_16=y # # AT91 Feature Selections Can accessing the flash delay interrupts significantly, and may this switch here have impact on the delay? Sorry for potential stupid questions, I'm not familiar with this arch yet. #./latency -t 2 -p 150 -h -H 500 ... ---|||||- RTS| 3.480| 11.779| 99.163| 0|14:23:01/14:23:01 It was run under calibrator load during more than 14 hours. Now RTAI: Oneshot timer with 500µs period, LATENCY =6000ns and SETUPTIME 1500 duration : 17h 1970/01/1 22:34:51 RTH|lat min|ovl min|lat avg|lat max|ovl max| overruns RTD| 3221| 2577| 4997| 26095| 53801| 0 RTD| 3221| 2577| 5163| 25451| 53801| 0 RTD| 3221| 2577| 5159| 25128| 53801| 0 RTD| 3221| 2577| 4799| 23518| 53801| 0 RTD| 3221| 2577| 4828| 25128| 53801| 0 RTD| 3221| 2577| 5089| 23518| 53801| 0 RTD| 3221| 2577| 4580| 23840| 53801| 0 RTD| 3221| 2577| 4924| 25128| 53801| 0 RTD| 3221| 2577| 4740| 24806| 53801| 0 RTD| 3221| 2577| 4251| 25128| 53801| 0 Again, what would simplify the discussion enormously is a function back-trace of the measured maximum latencies, just like latency -f generates. The numbers will become worse, for sure, but we would gain a very good overview about what is executed and what delayed which kernel. If you see a chance to perform such a test and you need some hints on the tracer setup (or did you use it before?), please let us know! OK for try it, at least for Xenomai as the function exists, but I can't do it tonight and not this week in fact. Thanks advance! BTW, you don't need to apply black magic in order to instrument RTAI: As you measure in the kernel anyway, you just need to add an ipipe_trace_freeze(measured_latency) at the point where a new maximum latency is detected in RTAI's benchmark module. In theory, the tracer should also work over RTAI out-of-the-box (but I'm not aware of anyone actually using it there). More info on the tracer can be found in the Xenomai wiki, BTW. Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list
Re: [Xenomai-core] Fwd: RTAI and Xenomai latency in kernel mode on AT91SAM9261-EK
Gregory CLEMENT wrote: first Xenomai: #insmod /lib/modules/2.6.20.13/kernel/drivers/xenomai/testing/xeno_timerbench.ko #cd /usr/xenomai/bin/ Which versions were you using for both tests? Do you still have the involved .configs? Both version are 2.6.20.13 I join the config files. BTW, the attached Xenomai config is obviously not the one that was used for the test: # # Testing drivers # # CONFIG_XENO_DRIVERS_TIMERBENCH is not set Jan signature.asc Description: OpenPGP digital signature ___ Xenomai-core mailing list Xenomai-core@gna.org https://mail.gna.org/listinfo/xenomai-core
