Ralf Ramsauer <[email protected]> writes: > On 04/05/2017 06:23 PM, Måns Rullgård wrote: >> Ralf Ramsauer <[email protected]> writes: >> >>> On 04/05/2017 02:48 PM, Jan Kiszka wrote: >>>> On 2017-04-05 13:43, Måns Rullgård wrote: >>>>> Ralf Ramsauer <[email protected]> writes: >>>>> >>>>>> Hi, >>>>>> >>>>>> I need spinlocks in inmates on ARM, so I simply included asm/spinlock.h. >>>>>> Taking locks in inmates on ARM somehow doesn't work at all, but that's >>>>>> not the main issue. As soon as I try to take locks, cells can not be >>>>>> destroyed any longer, the whole system freezes on cell destroy. >>>>>> >>>>>> I was able to trace this down to the 'ldrex' instruction. This minimal >>>>>> example triggers this behavior (tested on a TK1, can not test it on >>>>>> other boards): >>>>>> >>>>>> #include <inmate.h> >>>>>> >>>>>> void inmate_main(void) >>>>>> { >>>>>> unsigned int foo = 0, bar = 0; >>>>>> >>>>>> printk("Foo!\n"); >>>>>> asm volatile("ldrex %0, [%1]\n\t" >>>>>> : "=&r" (foo) : "r" (bar)); >>>>>> >>>>>> printk("Bar!\n"); >>>>>> while(1); >>>>>> asm volatile("wfi"); >>>>>> } >>>>>> >>>>>> The example code successfully finishes, I can read Foo and Bar on my >>>>>> serial console. But I am not able to destroy the cell afterwards. My >>>>>> system completely hangs on "jailhouse cell destroy". >>>>>> >>>>>> Curiously the cell can be re-loaded and started over again. Leaving out >>>>>> the ldrex instruction does not trigger the bug. >>>>>> >>>>>> - How can ldrex prevent my cell from being destroyed? >>>>>> >>>>>> Does it change some 'internal state' where jailhouse is not aware >>>>>> about? >>>> >>>> Is your inmate opting-out from the Comm Region protocol >>>> (JAILHOUSE_CELL_PASSIVE_COMMREG)? >>> Yes I'm using the Jailhouse's TK1 default config. >>>> >>>>>> >>>>>> - Why do spinlocks not work at all in inmates? >>>>>> >>>>>> spin_lock(&lock) never returns in inmates on a fresh spin lock. >>>>>> Does the ARM implementation of spinlocks require some special >>>>>> global initialisation that I don't know of? >>>>> >>>>> Spinlocks use ldrex/strex pairs, so if ldrex is broken, I wouldn't >>>>> expect spinlocks to work either. >>>>> >>>> >>>> ...and Linux use them heavily. All our ARM SMP Linux inmates would be >>>> broken then. There must be more involved. >>> Yes, true. >>> >>> I just reanimated my Orange Pi zero and cross-checked the behaviour there: >>> >>> It just prints the "Foo!", no bar, and I'm able to destroy its cell >>> afterwards. Jetson TK1 behaves different and freezes. >> >> Is that using the same compiled inmate code? Could you post a >> disassembly of the inmate_main() function? The compiler might be doing >> something unexpected. > Sure. Let me add some more detail. > > The source: > void inmate_main(void) > { > unsigned int foo = 0, bar = 0; > printk("Foo!\n"); > > asm volatile("dsb sy\n" ::: "memory"); > asm volatile( > "ldrex %0, [%1]\n\t" > // doesn't matter if I choose &bar > : "=&r" (foo) : "r" (bar) > ); > asm volatile("dsb sy\n" ::: "memory"); > > printk("Bar!\n"); > > while(1) > asm volatile("wfi"); > } > > compiles to: > 00000204 <inmate_main>: > 204: e92d4010 push {r4, lr} > 208: e59f0020 ldr r0, [pc, #32] ; 230 <inmate_main+0x2c> > 20c: eb0000ea bl 5bc <printk> > 210: f57ff04f dsb sy > 214: e3a03000 mov r3, #0 > 218: e1932f9f ldrex r2, [r3] > 21c: f57ff04f dsb sy > 220: e59f000c ldr r0, [pc, #12] ; 234 <inmate_main+0x30> > 224: eb0000e4 bl 5bc <printk> > 228: e320f003 wfi > 22c: eafffffd b 228 <inmate_main+0x24> > 230: 00000d79 .word 0x00000d79 > 234: 00000d7f .word 0x00000d7f
Nothing strange there. > The resulting binary can be used for both systems, I just have to align > some settings for the UART, which can be done per cmdline parameter. > > On TK1 side, it results in: > Cell "jetson-tk1-demo" can be loaded > Started cell "jetson-tk1-demo" > Foo! > Bar! > > When I try to 'jailhouse cell destroy jetson-tk1-demo' the last things I > get to see are: > [...] > Closing cell "jetson-tk1-demo" > Page pool usage after cell destruction: mem 71/16107, remap 69/131072 > > And that's it, Linux hung up, but *only* if the inmate executes the > ldrex instruction since the last full system reset. > > The very same inmate binary on the Orange Pi Zero (sorry, UARTs are shared): > root@orange:~/jailhouse# ./tools/jailhouse cell start 1 > Started cell "orangepi0-gic-demo" > Foo! > root@orange:~/jailhouse# ./tools/jailhouse disable > Closing cell "orangepi0-gic-demo" > Page pool usage after cell destruction: mem 36/2025, remap 69/131072 > [ 188.867422] Destroyed Jailhouse cell "orangepi0-gic-demo" > Shutting down hypervisor > Releasing CPU 2 > Releasing CPU 1 > Releasing CPU 3 > Releasing CPU 0 > [ 188.882025] The Jailhouse was closed. > > So on the Orangepi I can only see the "Foo!", and destroy the cell or > disable the hypervisor, whatever, and on the TK1 I see "Foo!\nBar!" and > the system freezes, as soon as I try to destroy the cell. What happens if you use a plain LDR instead of LDREX? -- Måns Rullgård -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
