On 04/05/2017 06:23 PM, Måns Rullgård wrote:
> Ralf Ramsauer <[email protected]> writes:
>
>> On 04/05/2017 02:48 PM, Jan Kiszka wrote:
>>> On 2017-04-05 13:43, Måns Rullgård wrote:
>>>> Ralf Ramsauer <[email protected]> writes:
>>>>
>>>>> Hi,
>>>>>
>>>>> I need spinlocks in inmates on ARM, so I simply included asm/spinlock.h.
>>>>> Taking locks in inmates on ARM somehow doesn't work at all, but that's
>>>>> not the main issue. As soon as I try to take locks, cells can not be
>>>>> destroyed any longer, the whole system freezes on cell destroy.
>>>>>
>>>>> I was able to trace this down to the 'ldrex' instruction. This minimal
>>>>> example triggers this behavior (tested on a TK1, can not test it on
>>>>> other boards):
>>>>>
>>>>> #include <inmate.h>
>>>>>
>>>>> void inmate_main(void)
>>>>> {
>>>>> unsigned int foo = 0, bar = 0;
>>>>>
>>>>> printk("Foo!\n");
>>>>> asm volatile("ldrex %0, [%1]\n\t"
>>>>> : "=&r" (foo) : "r" (bar));
>>>>>
>>>>> printk("Bar!\n");
>>>>> while(1);
>>>>> asm volatile("wfi");
>>>>> }
>>>>>
>>>>> The example code successfully finishes, I can read Foo and Bar on my
>>>>> serial console. But I am not able to destroy the cell afterwards. My
>>>>> system completely hangs on "jailhouse cell destroy".
>>>>>
>>>>> Curiously the cell can be re-loaded and started over again. Leaving out
>>>>> the ldrex instruction does not trigger the bug.
>>>>>
>>>>> - How can ldrex prevent my cell from being destroyed?
>>>>>
>>>>> Does it change some 'internal state' where jailhouse is not aware
>>>>> about?
>>>
>>> Is your inmate opting-out from the Comm Region protocol
>>> (JAILHOUSE_CELL_PASSIVE_COMMREG)?
>> Yes I'm using the Jailhouse's TK1 default config.
>>>
>>>>>
>>>>> - Why do spinlocks not work at all in inmates?
>>>>>
>>>>> spin_lock(&lock) never returns in inmates on a fresh spin lock.
>>>>> Does the ARM implementation of spinlocks require some special
>>>>> global initialisation that I don't know of?
>>>>
>>>> Spinlocks use ldrex/strex pairs, so if ldrex is broken, I wouldn't
>>>> expect spinlocks to work either.
>>>>
>>>
>>> ...and Linux use them heavily. All our ARM SMP Linux inmates would be
>>> broken then. There must be more involved.
>> Yes, true.
>>
>> I just reanimated my Orange Pi zero and cross-checked the behaviour there:
>>
>> It just prints the "Foo!", no bar, and I'm able to destroy its cell
>> afterwards. Jetson TK1 behaves different and freezes.
>
> Is that using the same compiled inmate code? Could you post a
> disassembly of the inmate_main() function? The compiler might be doing
> something unexpected.
Sure. Let me add some more detail.
The source:
void inmate_main(void)
{
unsigned int foo = 0, bar = 0;
printk("Foo!\n");
asm volatile("dsb sy\n" ::: "memory");
asm volatile(
"ldrex %0, [%1]\n\t"
// doesn't matter if I choose &bar
: "=&r" (foo) : "r" (bar)
);
asm volatile("dsb sy\n" ::: "memory");
printk("Bar!\n");
while(1)
asm volatile("wfi");
}
compiles to:
00000204 <inmate_main>:
204: e92d4010 push {r4, lr}
208: e59f0020 ldr r0, [pc, #32] ; 230 <inmate_main+0x2c>
20c: eb0000ea bl 5bc <printk>
210: f57ff04f dsb sy
214: e3a03000 mov r3, #0
218: e1932f9f ldrex r2, [r3]
21c: f57ff04f dsb sy
220: e59f000c ldr r0, [pc, #12] ; 234 <inmate_main+0x30>
224: eb0000e4 bl 5bc <printk>
228: e320f003 wfi
22c: eafffffd b 228 <inmate_main+0x24>
230: 00000d79 .word 0x00000d79
234: 00000d7f .word 0x00000d7f
The resulting binary can be used for both systems, I just have to align
some settings for the UART, which can be done per cmdline parameter.
On TK1 side, it results in:
Cell "jetson-tk1-demo" can be loaded
Started cell "jetson-tk1-demo"
Foo!
Bar!
When I try to 'jailhouse cell destroy jetson-tk1-demo' the last things I
get to see are:
[...]
Closing cell "jetson-tk1-demo"
Page pool usage after cell destruction: mem 71/16107, remap 69/131072
And that's it, Linux hung up, but *only* if the inmate executes the
ldrex instruction since the last full system reset.
The very same inmate binary on the Orange Pi Zero (sorry, UARTs are shared):
root@orange:~/jailhouse# ./tools/jailhouse cell start 1
Started cell "orangepi0-gic-demo"
Foo!
root@orange:~/jailhouse# ./tools/jailhouse disable
Closing cell "orangepi0-gic-demo"
Page pool usage after cell destruction: mem 36/2025, remap 69/131072
[ 188.867422] Destroyed Jailhouse cell "orangepi0-gic-demo"
Shutting down hypervisor
Releasing CPU 2
Releasing CPU 1
Releasing CPU 3
Releasing CPU 0
[ 188.882025] The Jailhouse was closed.
So on the Orangepi I can only see the "Foo!", and destroy the cell or
disable the hypervisor, whatever, and on the TK1 I see "Foo!\nBar!" and
the system freezes, as soon as I try to destroy the cell.
Ralf
>
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
