On 04/05/2017 07:05 PM, Måns Rullgård wrote: > Ralf Ramsauer <[email protected]> writes: > >> On 04/05/2017 06:23 PM, Måns Rullgård wrote: >>> Ralf Ramsauer <[email protected]> writes: >>> >>>> On 04/05/2017 02:48 PM, Jan Kiszka wrote: >>>>> On 2017-04-05 13:43, Måns Rullgård wrote: >>>>>> Ralf Ramsauer <[email protected]> writes: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I need spinlocks in inmates on ARM, so I simply included asm/spinlock.h. >>>>>>> Taking locks in inmates on ARM somehow doesn't work at all, but that's >>>>>>> not the main issue. As soon as I try to take locks, cells can not be >>>>>>> destroyed any longer, the whole system freezes on cell destroy. >>>>>>> >>>>>>> I was able to trace this down to the 'ldrex' instruction. This minimal >>>>>>> example triggers this behavior (tested on a TK1, can not test it on >>>>>>> other boards): >>>>>>> >>>>>>> #include <inmate.h> >>>>>>> >>>>>>> void inmate_main(void) >>>>>>> { >>>>>>> unsigned int foo = 0, bar = 0; >>>>>>> >>>>>>> printk("Foo!\n"); >>>>>>> asm volatile("ldrex %0, [%1]\n\t" >>>>>>> : "=&r" (foo) : "r" (bar)); >>>>>>> >>>>>>> printk("Bar!\n"); >>>>>>> while(1); >>>>>>> asm volatile("wfi"); >>>>>>> } >>>>>>> >>>>>>> The example code successfully finishes, I can read Foo and Bar on my >>>>>>> serial console. But I am not able to destroy the cell afterwards. My >>>>>>> system completely hangs on "jailhouse cell destroy". >>>>>>> >>>>>>> Curiously the cell can be re-loaded and started over again. Leaving out >>>>>>> the ldrex instruction does not trigger the bug. >>>>>>> >>>>>>> - How can ldrex prevent my cell from being destroyed? >>>>>>> >>>>>>> Does it change some 'internal state' where jailhouse is not aware >>>>>>> about? >>>>> >>>>> Is your inmate opting-out from the Comm Region protocol >>>>> (JAILHOUSE_CELL_PASSIVE_COMMREG)? >>>> Yes I'm using the Jailhouse's TK1 default config. >>>>> >>>>>>> >>>>>>> - Why do spinlocks not work at all in inmates? >>>>>>> >>>>>>> spin_lock(&lock) never returns in inmates on a fresh spin lock. >>>>>>> Does the ARM implementation of spinlocks require some special >>>>>>> global initialisation that I don't know of? >>>>>> >>>>>> Spinlocks use ldrex/strex pairs, so if ldrex is broken, I wouldn't >>>>>> expect spinlocks to work either. >>>>>> >>>>> >>>>> ...and Linux use them heavily. All our ARM SMP Linux inmates would be >>>>> broken then. There must be more involved. >>>> Yes, true. >>>> >>>> I just reanimated my Orange Pi zero and cross-checked the behaviour there: >>>> >>>> It just prints the "Foo!", no bar, and I'm able to destroy its cell >>>> afterwards. Jetson TK1 behaves different and freezes. >>> >>> Is that using the same compiled inmate code? Could you post a >>> disassembly of the inmate_main() function? The compiler might be doing >>> something unexpected. >> Sure. Let me add some more detail. >> >> The source: >> void inmate_main(void) >> { >> unsigned int foo = 0, bar = 0; >> printk("Foo!\n"); >> >> asm volatile("dsb sy\n" ::: "memory"); >> asm volatile( >> "ldrex %0, [%1]\n\t" >> // doesn't matter if I choose &bar >> : "=&r" (foo) : "r" (bar) >> ); >> asm volatile("dsb sy\n" ::: "memory"); >> >> printk("Bar!\n"); >> >> while(1) >> asm volatile("wfi"); >> } >> >> compiles to: >> 00000204 <inmate_main>: >> 204: e92d4010 push {r4, lr} >> 208: e59f0020 ldr r0, [pc, #32] ; 230 <inmate_main+0x2c> >> 20c: eb0000ea bl 5bc <printk> >> 210: f57ff04f dsb sy >> 214: e3a03000 mov r3, #0 >> 218: e1932f9f ldrex r2, [r3] >> 21c: f57ff04f dsb sy >> 220: e59f000c ldr r0, [pc, #12] ; 234 <inmate_main+0x30> >> 224: eb0000e4 bl 5bc <printk> >> 228: e320f003 wfi >> 22c: eafffffd b 228 <inmate_main+0x24> >> 230: 00000d79 .word 0x00000d79 >> 234: 00000d7f .word 0x00000d7f > > Nothing strange there. > >> The resulting binary can be used for both systems, I just have to align >> some settings for the UART, which can be done per cmdline parameter. >> >> On TK1 side, it results in: >> Cell "jetson-tk1-demo" can be loaded >> Started cell "jetson-tk1-demo" >> Foo! >> Bar! >> >> When I try to 'jailhouse cell destroy jetson-tk1-demo' the last things I >> get to see are: >> [...] >> Closing cell "jetson-tk1-demo" >> Page pool usage after cell destruction: mem 71/16107, remap 69/131072 >> >> And that's it, Linux hung up, but *only* if the inmate executes the >> ldrex instruction since the last full system reset. >> >> The very same inmate binary on the Orange Pi Zero (sorry, UARTs are shared): >> root@orange:~/jailhouse# ./tools/jailhouse cell start 1 >> Started cell "orangepi0-gic-demo" >> Foo! >> root@orange:~/jailhouse# ./tools/jailhouse disable >> Closing cell "orangepi0-gic-demo" >> Page pool usage after cell destruction: mem 36/2025, remap 69/131072 >> [ 188.867422] Destroyed Jailhouse cell "orangepi0-gic-demo" >> Shutting down hypervisor >> Releasing CPU 2 >> Releasing CPU 1 >> Releasing CPU 3 >> Releasing CPU 0 >> [ 188.882025] The Jailhouse was closed. >> >> So on the Orangepi I can only see the "Foo!", and destroy the cell or >> disable the hypervisor, whatever, and on the TK1 I see "Foo!\nBar!" and >> the system freezes, as soon as I try to destroy the cell. > > What happens if you use a plain LDR instead of LDREX? Everything as expected (TK1, didn't test the Orange, I expect the same):
Created cell "jetson-tk1-demo" Page pool usage after cell creation: mem 83/16107, remap 69/131072 Cell "jetson-tk1-demo" can be loaded Started cell "jetson-tk1-demo" Foo! Bar! Closing cell "jetson-tk1-demo" WARNING: Overflow during MMIO region registration! Page pool usage after cell destruction: mem 71/16107, remap 69/131072 Shutting down hypervisor Releasing CPU 1 Releasing CPU 0 Releasing CPU 3 Releasing CPU 2 So no freezes when trying to destroy the cell / disable jailhouse. Ralf -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
