Made the root cell to be 4.8.2-rt as well. With Jailhouse on, it runs the same test with typical jitter of 2-3uS and worse case of 35uS.
COUNTER SUM PER SEC vmexits_total 445327 1136 vmexits_virt_irq 430071 1107 vmexits_mmio 10073 15 vmexits_virt_sgi 4552 8 vmexits_hypercall 623 7 vmexits_management 14 0 vmexits_cp15 0 0 vmexits_maintenance 0 0 vmexits_psci 0 0 - Is the root cell handled differently on IRQ handling then non-root? - Why aren't there any CP15 exits, while the non-root cell with the same kernel, runs 3.5K/sec on timer rearming? And now the interesting part: *If the root cell itself* runs real-time 1ms task, the non-root cell average jitter drops to 12uS (from 50) and worth case to 70uS. How about that for a puzzle? On 24 February 2017 at 14:00, Dan Zach <[email protected]> wrote: > The CP15 exits are due to reprogramming the arm native timer via cp15 each > time. > > > On 24 February 2017 at 13:09, Dan Zach <[email protected]> wrote: > >> Hello Ralf. >> >> 1. Using just 4.8.2 with RT patch (without jailhouse) on Jetson TK1, I >> get worst case of 50uS with average of just 5uS. >> 2. The non-root linux: 1000HZ, applied the patch to timer.c (thanks for >> that) - still average delay on the same test is 40 uS and worth case goes >> up to 500uS >> >> So I see the steady state being x10 factor on the inmate, even without >> GPU or any stress at all. >> >> Not sure what the cp15exits mean here, but the number is huge (3.5K/sec) >> >> vmexits_total 3089789 3447 >> vmexits_cp15 2277030 2446 >> vmexits_virt_irq 812541 1000 >> vmexits_mmio 214 0 >> vmexits_psci 3 0 >> vmexits_management 2 0 >> vmexits_hypercall 0 0 >> vmexits_maintenance 0 0 >> vmexits_virt_sgi 0 0 >> >> >> >> On 23 February 2017 at 21:55, Ralf Ramsauer < >> [email protected]> wrote: >> >>> Hi Dan, >>> >>> On 02/23/2017 10:54 AM, Dan Zach wrote: >>> > So /proc/timer_list is below. >>> > In the dts configured both: the ARM internal timer and the SoC Tegra >>> > timer(dont think it actually works though): >>> > >>> > timer@0,60005000 { >>> > compatible = "nvidia,tegra124-timer", >>> > "nvidia,tegra20-timer"; >>> > reg = <0x60005000 0x400>; >>> > interrupts = <GIC_SPI 0 IRQ_TYPE_LEVEL_HIGH>, >>> > <GIC_SPI 1 IRQ_TYPE_LEVEL_HIGH>, >>> > <GIC_SPI 41 IRQ_TYPE_LEVEL_HIGH>, >>> > <GIC_SPI 42 IRQ_TYPE_LEVEL_HIGH>, >>> > <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>, >>> > <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>; >>> > clocks = <&tegra_car 5>; >>> > }; >>> Don't use this one. It requires a clock to be gated when being enabled, >>> so you also need to root-share the whole Tegra CAR (which doesn't work). >>> Partitioning of clock controllers is currently not supported by >>> Jailhouse. >>> >>> I've developed a paravirtual c&r controller for jailhouse, and we >>> already had some off-list discussions as we definitely will have to >>> address these issues in future, but for the moment clock gating is not >>> supported. >>> > >>> > >>> > timer { >>> > compatible = "arm,armv7-timer"; >>> > interrupts = <GIC_PPI 13 >>> > (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>, >>> > <GIC_PPI 14 >>> > (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>, >>> > <GIC_PPI 11 >>> > (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>, >>> > <GIC_PPI 10 >>> > (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_LOW)>; >>> > }; >>> Yep, use this timer. Should be absolutely sufficient. >>> > >>> > >>> > Can you tell if below looks ok? >>> Ok, so this is the timer list of the non-root RT Linux, right? >>> >>> Does you non-root RT Linux contain this [1] patch? >>> >>> Ralf >>> >>> [1] >>> http://git.kiszka.org/?p=linux.git;a=commitdiff;h=70671bb3cd >>> 47fe70b9a7076625e09d0411d58de9 >>> > >>> > Timer List Version: v0.8 >>> > HRTIMER_MAX_CLOCK_BASES: 4 >>> > now at 145981795048 nsecs >>> > >>> > cpu: 0 >>> > clock 0: >>> > .base: ceec8980 >>> > .index: 0 >>> > .resolution: 1 nsecs >>> > .get_time: ktime_get >>> > .offset: 0 nsecs >>> > active timers: >>> > #0: <ceec8c50>, tick_sched_timer, S:01, tick_nohz_restart, swapper/0/0 >>> > # expires at 145982000000-145982000000 nsecs [in 204952 to 204952 >>> nsecs] >>> > #1: <cdc25e90>, hrtimer_wakeup, S:01, schedule_hrtimeout_range_clock, >>> > bus_10ms/40 >>> > # expires at 145982000000-145982000001 nsecs [in 204952 to 204953 >>> nsecs] >>> > #2: def_rt_bandwidth, sched_rt_period_timer, S:01, enqueue_task_rt, >>> > ktimersoftd/0/4 >>> > # expires at 146026000000-146026000000 nsecs [in 44204952 to 44204952 >>> > nsecs] >>> > #3: <ceec8d70>, watchdog_timer_fn, S:01, watchdog_enable, >>> watchdog/0/14 >>> > # expires at 148032000000-148032000000 nsecs [in 2050204952 to >>> > 2050204952 nsecs] >>> > 0 d_clock_timer, sched_clock_poll, S:01, sched_clock_postinit, >>> > swapper/0/--More-- >>> > 48 nsecs]s at 4398046511096-4398046511096 nsecs [in 4252064716048 to >>> > 42520647160--More-- >>> > clock 1: >>> > .base: ceec89c0 >>> > .index: 1 >>> > .resolution: 1 nsecs >>> > .get_time: ktime_get_real >>> > .offset: 0 nsecs >>> > active timers: >>> > clock 2: >>> > .base: ceec8a00 >>> > .index: 2 >>> > .resolution: 1 nsecs >>> > .get_time: ktime_get_boottime >>> > .offset: 0 nsecs >>> > active timers: >>> > clock 3: >>> > .base: ceec8a40 >>> > .index: 3 >>> > .resolution: 1 nsecs >>> > .get_time: ktime_get_clocktai >>> > .offset: 0 nsecs >>> > active timers: >>> > .expires_next : 145985000000 nsecs >>> > .hres_active : 1 .idle_jiffies : 4294667614 >>> > .idle_calls : 4 >>> > .idle_sleeps : 2 >>> > .idle_entrytime : 319965499 nsecs >>> > .idle_waketime : 319965499 nsecs >>> > .idle_exittime : 319985583 nsecs >>> > .idle_sleeptime : 1838332 nsecs >>> > .iowait_sleeptime: 0 nsecs >>> > .last_jiffies : 4294667615 >>> > .next_timer : 329000000 >>> > .idle_expires : 329000000 nsecs >>> > jiffies: 4294813280 >>> > >>> > Tick Device: mode: 1 >>> > Broadcast device >>> > Clock Event Device: timer0 >>> > max_delta_ns: 536870948001 >>> > min_delta_ns: 1001 >>> > mult: 4294967 >>> > shift: 32 >>> > mode: 1 >>> > next_event: 9223372036854775807 nsecs >>> > set_next_event: tegra_timer_set_next_event >>> > shutdown: tegra_timer_shutdown >>> > periodic: tegra_timer_set_periodic >>> > oneshot: tegra_timer_shutdown >>> > resume: tegra_timer_shutdown >>> > event_handler: tick_handle_oneshot_broadcast >>> > retries: 0 >>> > >>> > tick_broadcast_mask: 0 >>> > tick_broadcast_oneshot_mask: 0 >>> > >>> > Tick Device: mode: 1 >>> > Per CPU device: 0 >>> > Clock Event Device: arch_sys_timer >>> > max_delta_ns: 178956969028 >>> > min_delta_ns: 1250 >>> > mult: 51539608 >>> > shift: 32 >>> > mode: 3 >>> > next_event: 145985000000 nsecs >>> > set_next_event: arch_timer_set_next_event_virt >>> > shutdown: arch_timer_shutdown_virt >>> > oneshot stopped: arch_timer_shutdown_virt >>> > event_handler: hrtimer_interrupt >>> > retries: 4 >>> > >>> > # >>> > # >>> > >>> > .nr_events : 145926 >>> > .nr_retries : 1 >>> > .nr_hangs : 0 >>> > .max_hang_time : 0 >>> > .nohz_mode : 2 >>> > .last_tick : 319000000 nsecs >>> > .tick_stopped : 0 >>> > >>> > >>> > On 23 February 2017 at 19:22, Ralf Ramsauer >>> > <[email protected] >>> > <mailto:[email protected]>> wrote: >>> > >>> > On 02/23/2017 08:17 AM, Henning Schild wrote: >>> > > On Thu, 23 Feb 2017 06:00:15 -0800 >>> > > Dan Zach <[email protected] <mailto:[email protected]>> wrote: >>> > > >>> > >> Dear forum, >>> > >> >>> > >> On the Jetson TK1 inmate I use linux 4.8.2 with PREEMPT-RT >>> patch. >>> > >> I measure a 1KHz high priority task based on hrtimer events that >>> > >> measures it's own jitter. >>> > Run jailhouse cell stats for your RT-cell and watch the MMIO >>> traps. I >>> > guess it will heavily trap on RT. RT heavily accesses the GICD that >>> > needs to be emulated in hardware >>> > >> >>> > >> The worst case, I get 400uS jitter on 1mS tick - not so bad, >>> but the >>> > Uh, okay? With the root cell idling? CONFIG_HZ_1000? >>> > >>> > Uhm -- What's your clocksource? Could you please post >>> /proc/timer_list? >>> > >> interesting thing is that the jitter stays as low as 50uS, >>> untill I >>> > >> start activity on the root cell, especially GUI. >>> > >> >>> > >> So the questions is: if the non-root cell is completely >>> isolated from >>> > >> the root: separate physical memory, PPI from its local timer, >>> where >>> > >> this cross influence can come from? >>> > > >>> > > Well the cells are not completely isolated, some shared resources >>> > > remain. The jitter you are seeing is caused by those, i.e. >>> caches and >>> > > busses. And GPU workloads would likely stress exactly those. >>> > Yep, mainly shared system bus and caches. Others like MMIO >>> dispatching >>> > and IRQ reinjection cause (at least measurable) latencies. I >>> already did >>> > some measurements to get worst case latencies, but I never hit >>> anything >>> > like that, interestingly. >>> > >>> > Do those latencies also occur if you use an Preempt-RT patched >>> kernel >>> > without Jailhouse when the GPU gets stressed? >>> > >>> > Ralf >>> > > >>> > > Henning >>> > > >>> > >> Thanks >>> > >> Dan >>> > >> >>> > > >>> > >>> > -- >>> > You received this message because you are subscribed to a topic in >>> > the Google Groups "Jailhouse" group. >>> > To unsubscribe from this topic, visit >>> > https://groups.google.com/d/topic/jailhouse-dev/RfucfkcbNQU >>> /unsubscribe >>> > <https://groups.google.com/d/topic/jailhouse-dev/RfucfkcbNQ >>> U/unsubscribe>. >>> > To unsubscribe from this group and all its topics, send an email to >>> > [email protected] >>> > <mailto:jailhouse-dev%[email protected]>. >>> > For more options, visit https://groups.google.com/d/optout >>> > <https://groups.google.com/d/optout>. >>> > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups "Jailhouse" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> > an email to [email protected] >>> > <mailto:[email protected]>. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> >> > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
