Hi,
On 18/11/2022 02:19, Karim Manaouil wrote:
Hi Henning,
I spent some more time debugging the issue.
I am getting a "FATAL: Invalid MMIO/RAM write".
Probably it's happening right after the first cpu that
calls arch_cpu_activate_mm() in hypervisor/setup.c:entry().
Not sure why, but maybe you have some pointers.
Here is the jailhouse output copy-pasted below.
Cheers
Initializing Jailhouse hypervisor v0.12 (314-gc7a1b697-dirty) on CPU 6
Code location: 0xfffffffff0000050
Using xAPIC
Page pool usage after early setup: mem 813/7635, remap 1/131072
Initializing processors:
Ok, having read your log, you have (at the moment) at least two issues:
First:
> FATAL: Invalid PCI config write, device 04:00.0, reg: 0xb4, size: 2
For the moment, go to your config and allow write access to that
capability. I could maybe help you with this if you share your config.
Please also attach lspci -vvv.
I can send you the config diff, and in the long run, you need to
understand what the changes mean.
Second:
> FATAL: Unhandled MSR read: c0002001
That's MSR_AMD64_SMCA_MC0_STATUS. For the moment, disable Machine Check
Events (MCE) in your kernel config or add appropriate parameters to your
kernel to disable them.
Disable CONFIG_X86_MCE_{INTEL,AMD} in .config, or try adding mce=off to
your kernel parameters.
HTH,
Ralf
CPU 6... (APIC ID 48) OK
CPU 64... (APIC ID 1) OK
CPU 0... (APIC ID 0) OK
CPU 16... (APIC ID 2) OK
CPU 112... (APIC ID 7) OK
CPU 104... (APIC ID 13) OK
CPU 40... (APIC ID 12) OK
CPU 72... (APIC ID 9) OK
CPU 8... (APIC ID 8) OK
CPU 56... (APIC ID 14) OK
CPU 120... (APIC ID 15) OK
CPU 110... (APIC ID 61) OK
CPU 46... (APIC ID 60) OK
CPU 14... (APIC ID 56) OK
CPU 78... (APIC ID 57) OK
CPU 94... (APIC ID 59) OK
CPU 30... (APIC ID 58) OK
CPU 126... (APIC ID 63) OK
CPU 62... (APIC ID 62) OK
CPU 2... (APIC ID 16) OK
CPU 66... (APIC ID 17) OK
CPU 18... (APIC ID 18) OK
CPU 82... (APIC ID 19) OK
CPU 114... (APIC ID 23) OK
CPU 50... (APIC ID 22) OK
CPU 98... (APIC ID 21) OK
CPU 34... (APIC ID 20) OK
CPU 12... (APIC ID 40) OK
CPU 76... (APIC ID 41) OK
CPU 60... (APIC ID 46) OK
CPU 124... (APIC ID 47) OK
CPU 44... (APIC ID 44) OK
CPU 108... (APIC ID 45) OK
CPU 92... (APIC ID 43) OK
CPU 28... (APIC ID 42) OK
CPU 26... (APIC ID 26) OK
CPU 90... (APIC ID 27) OK
CPU 74... (APIC ID 25) OK
CPU 10... (APIC ID 24) OK
CPU 106... (APIC ID 29) OK
CPU 42... (APIC ID 28) OK
CPU 58... (APIC ID 30) OK
CPU 122... (APIC ID 31) OK
CPU 20... (APIC ID 34) OK
CPU 84... (APIC ID 35) OK
CPU 36... (APIC ID 36) OK
CPU 100... (APIC ID 37) OK
CPU 116... (APIC ID 39) OK
CPU 52... (APIC ID 38) OK
CPU 4... (APIC ID 32) OK
CPU 68... (APIC ID 33) OK
CPU 96... (APIC ID 5) OK
CPU 32... (APIC ID 4) OK
CPU 88... (APIC ID 11) OK
CPU 55... (APIC ID 118) OK
CPU 119... (APIC ID 119) OK
CPU 87... (APIC ID 115) OK
CPU 23... (APIC ID 114) OK
CPU 71... (APIC ID 113) OK
CPU 7... (APIC ID 112) OK
CPU 39... (APIC ID 116) OK
CPU 103... (APIC ID 117) OK
CPU 47... (APIC ID 124) OK
CPU 111... (APIC ID 125) OK
CPU 15... (APIC ID 120) OK
CPU 79... (APIC ID 121) OK
CPU 31... (APIC ID 122) OK
CPU 95... (APIC ID 123) OK
CPU 127... (APIC ID 127) OK
CPU 63... (APIC ID 126) OK
CPU 86... (APIC ID 51) OK
CPU 22... (APIC ID 50) OK
CPU 38... (APIC ID 52) OK
CPU 102... (APIC ID 53) OK
CPU 118... (APIC ID 55) OK
CPU 54... (APIC ID 54) OK
CPU 70... (APIC ID 49) OK
CPU 109... (APIC ID 109) OK
CPU 45... (APIC ID 108) OK
CPU 93... (APIC ID 107) OK
CPU 29... (APIC ID 106) OK
CPU 13... (APIC ID 104) OK
CPU 77... (APIC ID 105) OK
CPU 61... (APIC ID 110) OK
CPU 125... (APIC ID 111) OK
CPU 5... (APIC ID 96) OK
CPU 101... (APIC ID 101) OK
CPU 37... (APIC ID 100) OK
CPU 85... (APIC ID 99) OK
CPU 21... (APIC ID 98) OK
CPU 117... (APIC ID 103) OK
CPU 53... (APIC ID 102) OK
CPU 69... (APIC ID 97) OK
CPU 49... (APIC ID 70) OK
CPU 1... (APIC ID 64) OK
CPU 65... (APIC ID 65) OK
CPU 81... (APIC ID 67) OK
CPU 17... (APIC ID 66) OK
CPU 97... (APIC ID 69) OK
CPU 33... (APIC ID 68) OK
CPU 113... (APIC ID 71) OK
CPU 25... (APIC ID 74) OK
CPU 89... (APIC ID 75) OK
CPU 41... (APIC ID 76) OK
CPU 105... (APIC ID 77) OK
CPU 9... (APIC ID 72) OK
CPU 73... (APIC ID 73) OK
CPU 121... (APIC ID 79) OK
CPU 57... (APIC ID 78) OK
CPU 3... (APIC ID 80) OK
CPU 67... (APIC ID 81) OK
CPU 35... (APIC ID 84) OK
CPU 99... (APIC ID 85) OK
CPU 115... (APIC ID 87) OK
CPU 51... (APIC ID 86) OK
CPU 19... (APIC ID 82) OK
CPU 83... (APIC ID 83) OK
CPU 107... (APIC ID 93) OK
CPU 43... (APIC ID 92) OK
CPU 11... (APIC ID 88) OK
CPU 75... (APIC ID 89) OK
CPU 123... (APIC ID 95) OK
CPU 59... (APIC ID 94) OK
CPU 27... (APIC ID 90) OK
CPU 24... (APIC ID 10) OK
CPU 80... (APIC ID 3) OK
CPU 48... (APIC ID 6) OK
CPU 91... (APIC ID 91) OK
Initializing unit: AMD IOMMU
AMD IOMMU @0xa1700000/0x80000
Initializing unit: IOAPIC
Initializing unit: PCI
Adding PCI device 00:00.0 to cell "RootCell"
Adding PCI device 00:01.0 to cell "RootCell"
Adding PCI device 00:01.1 to cell "RootCell"
Adding PCI device 00:01.3 to cell "RootCell"
Adding PCI device 00:01.4 to cell "RootCell"
Adding PCI device 00:02.0 to cell "RootCell"
Adding PCI device 00:03.0 to cell "RootCell"
Adding PCI device 00:04.0 to cell "RootCell"
Adding PCI device 00:07.0 to cell "RootCell"
Adding PCI device 00:07.1 to cell "RootCell"
Adding PCI device 00:08.0 to cell "RootCell"
Adding PCI device 00:08.1 to cell "RootCell"
Adding PCI device 00:14.0 to cell "RootCell"
Adding PCI device 00:14.3 to cell "RootCell"
Adding PCI device 00:18.0 to cell "RootCell"
Adding PCI device 00:18.1 to cell "RootCell"
Adding PCI device 00:18.2 to cell "RootCell"
Adding PCI device 00:18.3 to cell "RootCell"
Adding PCI device 00:18.4 to cell "RootCell"
Adding PCI device 00:18.5 to cell "RootCell"
Adding PCI device 00:18.6 to cell "RootCell"
Adding PCI device 00:18.7 to cell "RootCell"
Adding PCI device 00:19.0 to cell "RootCell"
Adding PCI device 00:19.1 to cell "RootCell"
Adding PCI device 00:19.2 to cell "RootCell"
Adding PCI device 00:19.3 to cell "RootCell"
Adding PCI device 00:19.4 to cell "RootCell"
Adding PCI device 00:19.5 to cell "RootCell"
Adding PCI device 00:19.6 to cell "RootCell"
Adding PCI device 00:19.7 to cell "RootCell"
Adding PCI device 00:1a.0 to cell "RootCell"
Adding PCI device 00:1a.1 to cell "RootCell"
Adding PCI device 00:1a.2 to cell "RootCell"
Adding PCI device 00:1a.3 to cell "RootCell"
Adding PCI device 00:1a.4 to cell "RootCell"
Adding PCI device 00:1a.5 to cell "RootCell"
Adding PCI device 00:1a.6 to cell "RootCell"
Adding PCI device 00:1a.7 to cell "RootCell"
Adding PCI device 00:1b.0 to cell "RootCell"
Adding PCI device 00:1b.1 to cell "RootCell"
Adding PCI device 00:1b.2 to cell "RootCell"
Adding PCI device 00:1b.3 to cell "RootCell"
Adding PCI device 00:1b.4 to cell "RootCell"
Adding PCI device 00:1b.5 to cell "RootCell"
Adding PCI device 00:1b.6 to cell "RootCell"
Adding PCI device 00:1b.7 to cell "RootCell"
Adding PCI device 00:1c.0 to cell "RootCell"
Adding PCI device 00:1c.1 to cell "RootCell"
Adding PCI device 00:1c.2 to cell "RootCell"
Adding PCI device 00:1c.3 to cell "RootCell"
Adding PCI device 00:1c.4 to cell "RootCell"
Adding PCI device 00:1c.5 to cell "RootCell"
Adding PCI device 00:1c.6 to cell "RootCell"
Adding PCI device 00:1c.7 to cell "RootCell"
Adding PCI device 00:1d.0 to cell "RootCell"
Adding PCI device 00:1d.1 to cell "RootCell"
Adding PCI device 00:1d.2 to cell "RootCell"
Adding PCI device 00:1d.3 to cell "RootCell"
Adding PCI device 00:1d.4 to cell "RootCell"
Adding PCI device 00:1d.5 to cell "RootCell"
Adding PCI device 00:1d.6 to cell "RootCell"
Adding PCI device 00:1d.7 to cell "RootCell"
Adding PCI device 00:1e.0 to cell "RootCell"
Adding PCI device 00:1e.1 to cell "RootCell"
Adding PCI device 00:1e.2 to cell "RootCell"
Adding PCI device 00:1e.3 to cell "RootCell"
Adding PCI device 00:1e.4 to cell "RootCell"
Adding PCI device 00:1e.5 to cell "RootCell"
Adding PCI device 00:1e.6 to cell "RootCell"
Adding PCI device 00:1e.7 to cell "RootCell"
Adding PCI device 00:1f.0 to cell "RootCell"
Adding PCI device 00:1f.1 to cell "RootCell"
Adding PCI device 00:1f.2 to cell "RootCell"
Adding PCI device 00:1f.3 to cell "RootCell"
Adding PCI device 00:1f.4 to cell "RootCell"
Adding PCI device 00:1f.5 to cell "RootCell"
Adding PCI device 00:1f.6 to cell "RootCell"
Adding PCI device 00:1f.7 to cell "RootCell"
Adding PCI device 01:00.0 to cell "RootCell"
Adding PCI device 01:00.1 to cell "RootCell"
Adding PCI device 02:00.0 to cell "RootCell"
Adding PCI device 03:00.0 to cell "RootCell"
Adding PCI device 04:00.0 to cell "RootCell"
Adding PCI device 04:00.1 to cell "RootCell"
Adding PCI device 05:00.0 to cell "RootCell"
Adding PCI device 05:00.2 to cell "RootCell"
Adding PCI device 05:00.3 to cell "RootCell"
Adding PCI device 06:00.0 to cell "RootCell"
Adding PCI device 06:00.1 to cell "RootCell"
Adding PCI device 06:00.2 to cell "RootCell"
Adding PCI device 20:00.0 to cell "RootCell"
Adding PCI device 20:00.2 to cell "RootCell"
Adding PCI device 20:01.0 to cell "RootCell"
Adding PCI device 20:01.1 to cell "RootCell"
Adding PCI device 20:02.0 to cell "RootCell"
Adding PCI device 20:03.0 to cell "RootCell"
Adding PCI device 20:04.0 to cell "RootCell"
Adding PCI device 20:07.0 to cell "RootCell"
Adding PCI device 20:07.1 to cell "RootCell"
Adding PCI device 20:08.0 to cell "RootCell"
Adding PCI device 20:08.1 to cell "RootCell"
Adding PCI device 21:00.0 to cell "RootCell"
Adding PCI device 21:00.1 to cell "RootCell"
Adding PCI device 22:08.0 to cell "RootCell"
Adding PCI device 23:00.0 to cell "RootCell"
Adding PCI device 24:00.0 to cell "RootCell"
Adding PCI device 24:00.2 to cell "RootCell"
Adding PCI device 24:00.3 to cell "RootCell"
Adding PCI device 25:00.0 to cell "RootCell"
Adding PCI device 25:00.1 to cell "RootCell"
Adding PCI device 40:00.0 to cell "RootCell"
Adding PCI device 40:00.2 to cell "RootCell"
Adding PCI device 40:01.0 to cell "RootCell"
Adding PCI device 40:02.0 to cell "RootCell"
Adding PCI device 40:03.0 to cell "RootCell"
Adding PCI device 40:03.1 to cell "RootCell"
Adding PCI device 40:04.0 to cell "RootCell"
Adding PCI device 40:07.0 to cell "RootCell"
Adding PCI device 40:07.1 to cell "RootCell"
Adding PCI device 40:08.0 to cell "RootCell"
Adding PCI device 40:08.1 to cell "RootCell"
Adding PCI device 41:00.0 to cell "RootCell"
Adding PCI device 41:00.1 to cell "RootCell"
Adding PCI device 42:00.0 to cell "RootCell"
Adding PCI device 42:00.2 to cell "RootCell"
Adding PCI device 43:00.0 to cell "RootCell"
Adding PCI device 43:00.1 to cell "RootCell"
Adding PCI device 60:00.0 to cell "RootCell"
Adding PCI device 60:00.2 to cell "RootCell"
Adding PCI device 60:01.0 to cell "RootCell"
Adding PCI device 60:02.0 to cell "RootCell"
Adding PCI device 60:03.0 to cell "RootCell"
Adding PCI device 60:03.1 to cell "RootCell"
Adding PCI device 60:04.0 to cell "RootCell"
Adding PCI device 60:07.0 to cell "RootCell"
Adding PCI device 60:07.1 to cell "RootCell"
Adding PCI device 60:08.0 to cell "RootCell"
Adding PCI device 60:08.1 to cell "RootCell"
Adding PCI device 61:00.0 to cell "RootCell"
Adding PCI device 62:00.0 to cell "RootCell"
Adding PCI device 62:00.2 to cell "RootCell"
Adding PCI device 63:00.0 to cell "RootCell"
Adding PCI device 63:00.1 to cell "RootCell"
Adding PCI device 80:00.0 to cell "RootCell"
Adding PCI device 80:00.2 to cell "RootCell"
Adding PCI device 80:01.0 to cell "RootCell"
Adding PCI device 80:02.0 to cell "RootCell"
Adding PCI device 80:03.0 to cell "RootCell"
Adding PCI device 80:04.0 to cell "RootCell"
Adding PCI device 80:07.0 to cell "RootCell"
Adding PCI device 80:07.1 to cell "RootCell"
Adding PCI device 80:08.0 to cell "RootCell"
Adding PCI device 80:08.1 to cell "RootCell"
Adding PCI device 81:00.0 to cell "RootCell"
Adding PCI device 81:00.2 to cell "RootCell"
Adding PCI device 82:00.0 to cell "RootCell"
Adding PCI device 82:00.1 to cell "RootCell"
Adding PCI device a0:00.0 to cell "RootCell"
Adding PCI device a0:00.2 to cell "RootCell"
Adding PCI device a0:01.0 to cell "RootCell"
Adding PCI device a0:02.0 to cell "RootCell"
Adding PCI device a0:03.0 to cell "RootCell"
Adding PCI device a0:04.0 to cell "RootCell"
Adding PCI device a0:07.0 to cell "RootCell"
Adding PCI device a0:07.1 to cell "RootCell"
Adding PCI device a0:08.0 to cell "RootCell"
Adding PCI device a0:08.1 to cell "RootCell"
Adding PCI device a1:00.0 to cell "RootCell"
Adding PCI device a1:00.2 to cell "RootCell"
Adding PCI device a2:00.0 to cell "RootCell"
Adding PCI device a2:00.1 to cell "RootCell"
Adding PCI device c0:00.0 to cell "RootCell"
Adding PCI device c0:00.2 to cell "RootCell"
Adding PCI device c0:01.0 to cell "RootCell"
Adding PCI device c0:02.0 to cell "RootCell"
Adding PCI device c0:03.0 to cell "RootCell"
Adding PCI device c0:03.1 to cell "RootCell"
Adding PCI device c0:03.2 to cell "RootCell"
Adding PCI device c0:03.3 to cell "RootCell"
Adding PCI device c0:03.4 to cell "RootCell"
Adding PCI device c0:04.0 to cell "RootCell"
Adding PCI device c0:07.0 to cell "RootCell"
Adding PCI device c0:07.1 to cell "RootCell"
Adding PCI device c0:08.0 to cell "RootCell"
Adding PCI device c0:08.1 to cell "RootCell"
Adding PCI device c1:00.0 to cell "RootCell"
Adding PCI device c2:00.0 to cell "RootCell"
Adding PCI device c3:00.0 to cell "RootCell"
Adding PCI device c4:00.0 to cell "RootCell"
Adding PCI device c5:00.0 to cell "RootCell"
Adding PCI device c5:00.2 to cell "RootCell"
Adding PCI device c6:00.0 to cell "RootCell"
Adding PCI device c6:00.1 to cell "RootCell"
Adding PCI device e0:00.0 to cell "RootCell"
Adding PCI device e0:00.2 to cell "RootCell"
Adding PCI device e0:01.0 to cell "RootCell"
Adding PCI device e0:02.0 to cell "RootCell"
Adding PCI device e0:03.0 to cell "RootCell"
Adding PCI device e0:04.0 to cell "RootCell"
Adding PCI device e0:07.0 to cell "RootCell"
Adding PCI device e0:07.1 to cell "RootCell"
Adding PCI device e0:08.0 to cell "RootCell"
Adding PCI device e0:08.1 to cell "RootCell"
Adding PCI device e1:00.0 to cell "RootCell"
Adding PCI device e1:00.2 to cell "RootCell"
Adding PCI device e2:00.0 to cell "RootCell"
Adding PCI device e2:00.1 to cell "RootCell"
Page pool usage after late setup: mem 1927/7635, remap 65703/131072
FATAL: Invalid MMIO/RAM write, addr: 0x00000000a1702008 size: 4
RIP: 0xffffffffa79d7777 RSP: 0xffffa2f7cda78de0 FLAGS: 6
RAX: 0xffffa2f7c0080000 RBX: 0x0000000000000001 RCX: 0x0000000000000030
RDX: 0xffff90d18000a000 RSI: 0x3000001700000000 RDI: 0x7ffffffffffff003
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x000000014487c000 CR4: 0x00000000003506e0
EFER: 0x0000000000001d01
Parking CPU 76 (Cell: "RootCell")
Ignoring NMI IPI to CPU 1
Ignoring NMI IPI to CPU 2
Ignoring NMI IPI to CPU 3
Ignoring NMI IPI to CPU 4
Ignoring NMI IPI to CPU 5
Ignoring NMI IPI to CPU 6
Ignoring NMI IPI to CPU 7
Ignoring NMI IPI to CPU 76
Ignoring NMI IPI to CPU 1
Ignoring NMI IPI to CPU 2
Ignoring NMI IPI to CPU 3
Ignoring NMI IPI to CPU 4
Ignoring NMI IPI to CPU 5
Ignoring NMI IPI to CPU 6
Ignoring NMI IPI to CPU 7
Ignoring NMI IPI to CPU 76
Ignoring NMI IPI to CPU 1
Ignoring NMI IPI to CPU 2
Ignoring NMI IPI to CPU 3
Ignoring NMI IPI to CPU 4
Ignoring NMI IPI to CPU 5
FATAL: Invalid PCI config write, device 04:00.0, reg: 0xb4, size: 2
RIP: 0xffffffffa7c52b3d RSP: 0xffffa2f7ce99bd98 FLAGS: 46
RAX: 0x000000000000242e RBX: 0x0000000000000000 RCX: 0x00000000000000b4
RDX: 0x0000000000000cfc RSI: 0x0000000000000216 RDI: 0xffffffffa9401790
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x0000003b99810000 CR4: 0x00000000003506e0
EFER: 0x0000000000001d01
Parking CPU 24 (Cell: "RootCell")
Ignoring NMI IPI to CPU 6
Ignoring NMI IPI to CPU 7
FATAL: Unhandled MSR read: c0002001
RIP: 0xffffffffa7c951cd RSP: 0xffffa2f7cd918e08 FLAGS: 246
RAX: 0x00000000c0002000 RBX: 0xffff90e15fc11020 RCX: 0x00000000c0002001
RDX: 0x0000000000000000 RSI: 0xffffa2f7cd918df0 RDI: 0x00000000c0002001
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x00000018a6a46000 CR4: 0x00000000003506e0
EFER: 0x0000000000001d01
Parking CPU 68 (Cell: "RootCell")
FATAL: Unhandled MSR read: c0002001
RIP: 0xffffffffa7c951cd RSP: 0xffffa2f7cd4f8e08 FLAGS: 246
RAX: 0x00000000c0002000 RBX: 0xffff90e15fb51020 RCX: 0x00000000c0002001
RDX: 0x0000000000000000 RSI: 0xffffa2f7cd4f8df0 RDI: 0x00000000c0002001
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x0000002081eaa000 CR4: 0x00000000003506e0
EFER: 0x0000000000001d01
Parking CPU 44 (Cell: "RootCell")
------------------------------------------------------------------------
*From:* Henning Schild <[email protected]>
*Sent:* 14 November 2022 09:22
*To:* Karim Manaouil <[email protected]>
*Cc:* Ralf Ramsauer <[email protected]>;
[email protected] <[email protected]>;
[email protected] <[email protected]>
*Subject:* Re: Jailhouse: enter_hypervisor returns -ENOMEM
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that
the email is genuine and the content is safe.
Am Sun, 13 Nov 2022 22:24:45 +0000
schrieb Karim Manaouil <[email protected]>:
Hi Ralf,
Thanks for the reply!
>Did you use jailhouse-config-create?
I am using `jailhouse config create` to generate the sysconfig.c file.
>You can use the --mem-hv option to
increate the memory. Try, for example, 32MiB and see if it works.
I tried with 32MiB. It worked. I am not getting -ENOMEM anymore.
The driver prints "The Jailhouse is opening" on dmesg. However, right
after that the CPUs get stuck, and I get rcu_sched detected stalls.
The system is completely irresponsive.
I attached a text file containing the full output from dmesg. Here is
the initial part:
I guess the output of the hypervisor might also be valuable here.
According to its spec that machine should have a serial port, and with
that default config from the generate script you should see logs coming
out of there. With the usual 115200 8n1
Henning
[ 434.792008] The Jailhouse is opening.
[ 455.787315] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 455.793303] rcu: 1-...0: (839 GPs behind)
idle=c2a/1/0x4000000000000000 softirq=681/681 fqs=1827 [ 455.802292]
rcu: 2-...0: (144 GPs behind) idle=812/1/0x4000000000000000
softirq=905/905 fqs=1827 [ 455.811276] rcu: 3-...0: (144 GPs
behind) idle=eaa/1/0x4000000000000000 softirq=719/719 fqs=1827 [
455.820266] rcu: 4-...0: (1 GPs behind)
idle=c2e/1/0x4000000000000000 softirq=1324/1324 fqs=1827 [
455.829252] rcu: 5-...0: (144 GPs behind)
idle=41a/1/0x4000000000000000 softirq=556/556 fqs=1827 [ 455.838238]
rcu: 6-...0: (144 GPs behind) idle=912/1/0x4000000000000000
softirq=777/777 fqs=1827 [ 455.847218] rcu: 7-...0: (144 GPs
behind) idle=5e6/1/0x4000000000000000 softirq=2409/2410 fqs=1827 [
455.856404] (detected by 87, t=5253 jiffies, g=48537, q=364) [
455.862170] Sending NMI from CPU 87 to CPUs 1: [ 465.776884] Sending
NMI from CPU 87 to CPUs 2: [ 467.182686] watchdog: BUG: soft lockup
- CPU#0 stuck for 23s! [kworker/0:1:7] [ 467.189857] Modules linked
in: jailhouse(O) nf_conntrack_netlink xfrm_user xt_addrtype
br_netfilter xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT
nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_natp [
467.189928] binfmt_misc configfs efivarfs ip_tables x_tables autofs4
ext4 crc16 mbcache jbd2 raid10 raid456 libcrc32c crc32c_generic
async_raid6_recov async_memcpy async_pq async_xor xor async_tx
raid6_pq ] [ 467.320567] CPU: 0 PID: 7 Comm: kworker/0:1 Tainted: G
O 5.10.0 #3 [ 467.328767] Hardware name: Dell Inc.
PowerEdge R7425/08V001, BIOS 1.15.0 09/11/2020 [ 467.337154]
Workqueue: events drm_fb_helper_dirty_work [drm_kms_helper] [
467.344501] RIP: 0010:smp_call_function_many_cond+0x289/0x2d0 [
467.350979] Code: e8 1c 8a 39 00 3b 05 0a c1 74 01 89 c7 0f 83 0b fe
ff ff 48 63 c7 49 8b 16 48 03 14 c5 00 d9 99 9c 8b 42 08 a8 01 74 09
f3 90 <8b> 42 08 a8 01 75 f7 eb c9 48 c7 c2 20 cf 07 9d 4c 89 fe 44 7
[ 467.371232] RSP: 0018:ffffa7d78015fcd8 EFLAGS: 00000202 [
467.377220] RAX: 0000000000000011 RBX: 0000000000031280 RCX:
0000000000000001 [ 467.385123] RDX: ffff964f1fa31280 RSI:
0000000000000000 RDI: 0000000000000001 [ 467.393024] RBP:
0000000000000000 R08: 0000000000000000 R09: 0000000000000001 [
467.400928] R10: 0000000000000002 R11: 0000000000000002 R12:
0000000000000000 [ 467.408836] R13: 000000000000007f R14:
ffff962f1f42c9c0 R15: 0000000000000080 [ 467.416737] FS:
0000000000000000(0000) GS:ffff962f1f400000(0000)
knlGS:0000000000000000 [ 467.425604] CS: 0010 DS: 0000 ES: 0000
CR0: 0000000080050033 [ 467.432127] CR2: 0000000000000000 CR3:
00000010987ea000 CR4: 00000000003506f0 [ 467.440045] Call Trace: [
467.443289] ? tlbflush_read_file+0x70/0x70 [ 467.448266] ?
tlbflush_read_file+0x70/0x70 [ 467.453242] on_each_cpu+0x2b/0x60 [
467.457437] __purge_vmap_area_lazy+0x5d/0x680 [ 467.462679] ?
_cond_resched+0x16/0x40 [ 467.467224] ?
unmap_kernel_range_noflush+0x2fa/0x380 [ 467.473072]
free_vmap_area_noflush+0xe7/0x100 [ 467.478315]
remove_vm_area+0x96/0xa0 [ 467.482770] __vunmap+0x8d/0x290 [
467.486792] drm_gem_shmem_vunmap+0x8b/0xa0 [drm] [ 467.492299]
drm_client_buffer_vunmap+0x16/0x20 [drm] [ 467.498144]
drm_fb_helper_dirty_work+0x187/0x1b0 [drm_kms_helper] [ 467.505118]
process_one_work+0x1b6/0x350 [ 467.509912] worker_thread+0x53/0x3e0
[ 467.514361] ? process_one_work+0x350/0x350 [ 467.519338]
kthread+0x11b/0x140 [ 467.523342] ? __kthread_bind_mask+0x60/0x60 [
467.528389] ret_from_fork+0x22/0x30
Cheers
Karim
________________________________
From: Ralf Ramsauer <[email protected]>
Sent: 12 November 2022 17:47
To: Karim Manaouil <[email protected]>; [email protected]
<[email protected]> Cc: [email protected]
<[email protected]> Subject: Re: Jailhouse:
enter_hypervisor returns -ENOMEM
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that
the email is genuine and the content is safe.
On 12/11/2022 18:15, Karim Manaouil wrote:
> Hi Jan,
>
> I am trying to deploy Jailhouse on an AMD EPYC with 128 CPUs (8 NUMA
> nodes), running Linux kernel v5.10 (same version used by jailhouse
> CI with same patches applied).
>
> `jailhouse hardware check` return that everything is ok and that
> "Check passed!".
>
> Memory was reserved via `memmap=0x5200000$0x3a000000`
>
> However, enter_hypervisor() [1] fails when entry() is called on
> every cpu and return -ENOMEM as error_code.
Try to reserve more memory. Maybe the default size of 6MiB for HV
memory is insufficient for 128 CPUs.
Did you use jailhouse-config-create? You can use the --mem-hv option
to increate the memory. Try, for example, 32MiB and see if it works.
Ralf
>
> Do you possibly know where could the issue come from?
>
> Best
> Karim
>
> [1]
>
https://github.com/siemens/jailhouse/blob/c7a1b6971ac15e4be8a0918b9bef6e2cbd99f9fc/driver/main.c#L251
<https://github.com/siemens/jailhouse/blob/c7a1b6971ac15e4be8a0918b9bef6e2cbd99f9fc/driver/main.c#L251>
>
<https://github.com/siemens/jailhouse/blob/c7a1b6971ac15e4be8a0918b9bef6e2cbd99f9fc/driver/main.c#L251
<https://github.com/siemens/jailhouse/blob/c7a1b6971ac15e4be8a0918b9bef6e2cbd99f9fc/driver/main.c#L251>>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336. Is e buidheann
> carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba,
> àireamh clàraidh SC005336.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jailhouse" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to [email protected]
> <mailto:[email protected]
<mailto:[email protected]>>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB6018F1663ABE61DA3C697CA4A9039%40AM0PR05MB6018.eurprd05.prod.outlook.com
<https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB6018F1663ABE61DA3C697CA4A9039%40AM0PR05MB6018.eurprd05.prod.outlook.com>
>
<https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB6018F1663ABE61DA3C697CA4A9039%40AM0PR05MB6018.eurprd05.prod.outlook.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/AM0PR05MB6018F1663ABE61DA3C697CA4A9039%40AM0PR05MB6018.eurprd05.prod.outlook.com?utm_medium=email&utm_source=footer>>.
>
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/79cf1f00-9c20-404e-150b-42142c804d3e%40oth-regensburg.de.
