Hi, We did some performance measurements. Added counters in 3 places (per Irq source) 1. entry to jailhouse ISR 2. dispatch of interrupt to particular vector to particular core 3. in RTOS isr.
*We see that all 3 counters have the same value*, but we measure time gaps in RTOS in ISR invocation times, sometimes upto 60 uSec. It means that either a) interrupt request is lost. But, according to setup it is edge-triggered, it cannot be lost, just delayed. b) there is a delay of more than 60 usec in jailhouse ISR. questions: 1. Is it possible that jailhouse interrupt dispatching routine enters some loop that takes considerable amount of time? 2. What would be explanation of interrupt latency of 60 Secs? Even if we take into account cache line refill we get much lower number, which do not reach tens uSecs. Best regards Rasty On Tuesday, December 6, 2022 at 6:01:20 PM UTC+2 Ralf Ramsauer wrote: > Hi, > > On 05/12/2022 17:30, Rasty Slutsker wrote: > > Hi Ralf, > > Thank you for the answer. > > We have periodic interrupt each 30 u(!)Sec. Linux cannot deal with such > > rate, so we need hypervisor/RTOS. > > I understand. > > > We managed to read a code of hypervisor. It appears that all interrupts > > to all cores are intercepted by hypervisor and then forwarded to guests > > (per core). > > Yes, exactly, that's the case if you don't have SDEI. If you have a > platform that would come with SDEI, then you have of course less overhead. > > > If we reduce interrupt priority of mentioned interrupt (as you suggest) > > we lose even more interrupts, even without stress. > > Interrupt is defined as edge triggered, I assumed that it is memorized > > by gic until serviced. > > Is it possible that Hypervisor acknowledges pending interrupt while > > servicing interrupt from another source ? Kind of race - 2 interrupts > > for 2 cores arrive nearly simultaneously. One is lost. > > The EOIR and IAR registers of the GIC are core-local registers of the > GIC CPU interface (GICC), so I wonder how this should cause a race, > unless there isn't a hard logical mistake in the code, which I doubt. > > > What you could try to do for debugging purposes: > > 1. Slow down from 30µs to something sloooower, which you can handle > even under load. Say 100µs, 500µs, something like that. > > 2. Measure the jitter x between arrival of the interrupt, and final > acknowledgement in your RTOS. You can use performance monitoring > registers, or watch CPU cycle counters, whatever. Repeat the > measurement, w/ load and w/o load on Linux-side. > > 3. If max(x) >= 30µs, then you know where your IRQs go in case of a > periodic cycle of 30µs. > > > Reason: What I did some while ago, is measuring the Jitter of > Linux+Jailhouse on ARM systems with cyclictest. On a Jetson TX1 > platform, for example, we saw Jitter up 50µs. So there's IRQ > reinjection, a full Linux stack and some userspace application involved, > so three context switches and lots of code. You have probably two > context switches and less code, as you use a RTOS, but I think there's a > certain chance to exceed 30µs. > > What my gut feeling tells me is that you manage to hold those 30µs if > Linux is quiet. As soon as there's some stress on the system bus, and > even on shared caches, you exceed you deadline. > > Thanks > Ralf > > > > > Best regards > > Rasty > > > > On Monday, December 5, 2022 at 5:14:37 PM UTC+2 Ralf Ramsauer wrote: > > > > Hi Nir, > > > > On 29/11/2022 14:21, [email protected] wrote: > > > Hi there, > > > > > > Our target is Sitara AM5726 , CortexA15 dual core on which we are > > > running Linux on A15 core0 and RTOS on core1. > > > > > > __ > > > > > > RTOS gets periodic interrupt from external hardware via nirq1 pin > > > (dedicated input into ARM gic).____ > > > > > > Under heavy load in Linux (core 0!), RTOS, which runs on core1 > > misses > > > interrupts.____ > > > > Uhm. Can you reconstruct that issue w/o Jailhouse under Linux? > > > > I mean, can you set the SMP affinity of that IRQ to core 1 under Linux, > > and then write some test application running on core 1 that just > > receives the IRQ. If that issue happens under Linux as well, then you > > know that the issue has probably nothing to do with Jailhouse. > > > > > > What also might happen: If there's enough pressure on the shared system > > bus when Linux is under load, then you simply loose those IRQs as the > > RTOS doesn't have enough time to handle it. You can test this > > hypothesis > > if you lower the frequency of the the periodic interrupt. If you still > > loose IRQs, then this should not be the case. > > > > > > > > Questions____ > > > > > > 1. Does linux/hypervisor participate in interrupt > > scheduling/forwarding > > > to cell on Core1____ > > > > Linux: No, Linux does not participate in anything that is going on on > > CPU 1. That's the idea behind Jailhouse. > > > > Jailhouse: Maybe. On ARM platforms, Jailhouse needs to reinject the > > Interrupt from the hypervisor to the guest, if SDEI is not available. > > Does the Sitara come with support for SDEI support? > > > > (You can btw monitor the exits of the hypervisor with 'jailhouse cell > > stats') > > > > Ralf > > > > > 2. Is there a description of interrupt forwarding/virtualization > > scheme > > > to cores (if exists)? Any pointer to document/source code would be > > > appreciated. > > > > > > Thanks a lot, > > > > > > Nir. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "Jailhouse" group. > > > To unsubscribe from this group and stop receiving emails from it, > > send > > > an email to [email protected] > > > <mailto:[email protected]>. > > > To view this discussion on the web visit > > > > > > https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com > > < > https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com> > > < > https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com?utm_medium=email&utm_source=footer > > < > https://groups.google.com/d/msgid/jailhouse-dev/fde55f66-2e83-4df2-8f5e-44b0fb831acbn%40googlegroups.com?utm_medium=email&utm_source=footer > >>. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Jailhouse" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected] > > <mailto:[email protected]>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/jailhouse-dev/9f1616c7-ac5f-49de-bc24-8bd8520f4c07n%40googlegroups.com > > < > https://groups.google.com/d/msgid/jailhouse-dev/9f1616c7-ac5f-49de-bc24-8bd8520f4c07n%40googlegroups.com?utm_medium=email&utm_source=footer > >. > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/01dfb141-0e4f-4adc-939f-c7a0c145c055n%40googlegroups.com.
