Re: Need help to run Linux in non-root cell

Andrej Utz Wed, 08 Jul 2020 07:37:55 -0700

Hi all,

On 02/07/2020 18:26, Ralf Ramsauer wrote:



On 02/07/2020 18:17, Jan Kiszka wrote:

On 02.07.20 18:07, Moritz Walker wrote:

  Smells like a regression in that branch for non-root Linux. Is that
  562b04e51bb5e2f04bf175383080333237067c63? Can you share you kernel
config?


Yes, its 562b04e51bb5e2f04bf175383080333237067c63. I  attached the
kernel config.
I also tried the kernel from jailhouse-images (Linux version 5.4.17)
which leads to  the same error
on my AMD-machine:


Ah, AMD! Please see


Yeah, AMD. It isn't always only Intel. :-)

https://groups.google.com/forum/#!msg/jailhouse-dev/1wRKIiGN0GA/_p_NSIBpDwAJ
- in fact a known issue (hardware misbehavior) that we didn't workaround
yet.


Yes, just wanted to mention! Looks familiar.


Ralf, Andrej, any news here.


Andrej?
It's already been a while. Didn't we already have some preliminary
patches for Linux?

As a hacky workaround, you can try:

diff --git a/hypervisor/arch/x86/apic.c b/hypervisor/arch/x86/apic.c
index d36c2033..5160d37d 100644
--- a/hypervisor/arch/x86/apic.c
+++ b/hypervisor/arch/x86/apic.c
@@ -350,7 +350,7 @@ void apic_clear(void)

         /* Finally, reset the TPR again and disable the APIC */
         apic_ops.write(APIC_REG_TPR, 0);
-       apic_ops.write(APIC_REG_SVR, 0xff);
+       //apic_ops.write(APIC_REG_SVR, 0xff);
  }

  static void apic_send_ipi(unsigned int target_cpu_id, u32 orig_icr_hi,

Or as an alternative apply the following patch to the kernel:https://groups.google.com/d/msg/jailhouse-dev/1wRKIiGN0GA/P5YeS3oqAQAJ

@Jan: Any ideas how to bring the mentioned patch (or somethingequivalent) upstream? I know its a hack and will add proper CPU familychecks, but the linux-x86 folks still may not even touch it. MeanwhileAMD gains market share and as such more people will try to run non-rootLinux with their CPUs, run into this bug and request support here ordecide to use other hypervisors.


Andrej

>

   Ralf

Jan


[    5.879554] Switched APIC routing to physical flat.
FATAL: Setting invalid LVT delivery mode (reg 35, value 00000700)
FATAL: Unhandled APIC access, offset 848, is_write: 1

The jailhouse-images kernel (5.4.17) works fine on a different machine
(Intel).
Might this problem be realted to my first machine beeing an AMD-x86 one?

Moritz



------------------------------------------------------------------------
*Von:* Jan Kiszka <[email protected]>
*Gesendet:* Donnerstag, 2. Juli 2020 12:56
*An:* Moritz Walker <[email protected]>;
[email protected] <[email protected]>
*Betreff:* Re: Need help to run Linux in non-root cell
On 02.07.20 14:04, Moritz Walker wrote:

Hello,

I need help to run Linux (queues/jailhouse branch) in a non-root cell
on AMD. Root-cell and apic-demo both work fine.
To run the root-cell i run the /jailhouse enable
configs/x86/rootcell.cell/ comman, which produces the following output:


Initializing Jailhouse hypervisor v0.12 (59-g4ce7658d-dirty) on CPU 1
Code location: 0xfffffffff0000050
Using xAPIC
Page pool usage after early setup: mem 75/979, remap 1/131072
Initializing processors:
    CPU 1... (APIC ID 1) OK
[...]
    CPU 7... (APIC ID 5) OK
Initializing unit: AMD IOMMU
AMD IOMMU @0xfeb80000/0x80000
Initializing unit: IOAPIC
Initializing unit: PCI
[...]
Adding PCI device 31:00.0 to cell "RootCell"
Page pool usage after late setup: mem 286/979, remap 16520/131072
Activating hypervisor


After that i use the command /jailhouse cell linux
configs/x86/linux-x86-cell.cell ../linux/arch/x86/boot/bzImage -c
"console=ttyS0,9600"/
to start the non-root linux cell. This writes the following output
via UART:


Adding virtual PCI device 00:0c.0 to cell "linux-x86-demo"
Adding virtual PCI device 00:0d.0 to cell "linux-x86-demo"
Adding virtual PCI device 00:0e.0 to cell "linux-x86-demo"
Adding virtual PCI device 00:0f.0 to cell "linux-x86-demo"
Created cell "linux-x86-demo"
Page pool usage after cell creation: mem 343/979, remap 16520/131072
IOMMU 0: Event Log overflow occurred, some events were lost!
Cell "linux-x86-demo" can be loaded
Started cell "linux-x86-demo"
CPU 6 received SIPI, vector 100
CPU 7 received SIPI, vector 100
[    0.000000] Linux version 5.7.0-rc7+ (walker@wubuntu) (gcc version
9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2), GNU ld (GNU Binutils for
Ubuntu) 2.33) #2 SMP Wed Jul 1 12:28:55 CEST 2020
[    0.000000] Command line: console=ttyS0,9600
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating
point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is
832 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x00000000000fffff]
usable
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000000100fff]
reserved
[    0.000000] BIOS-e820: [mem 0x0000000000200000-0x00000000048fffff]
usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] extended physical RAM map:
[    0.000000] reserve setup_data: [mem
0x0000000000000000-0x0000000000001fff] usable
[    0.000000] reserve setup_data: [mem
0x0000000000002000-0x000000000000212b] usable
[    0.000000] reserve setup_data: [mem
0x000000000000212c-0x00000000000fffff] usable
[    0.000000] reserve setup_data: [mem
0x0000000000100000-0x0000000000100fff] reserved
[    0.000000] reserve setup_data: [mem
0x0000000000200000-0x00000000048fffff] usable
[    0.000000] DMI not present or invalid.
[    0.000000] Hypervisor detected: Jailhouse
[    0.000000] tsc: Detected 3599.999 MHz processor
[    0.000016] last_pfn = 0x4900 max_arch_pfn = 0x400000000
[    0.000863] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP
  UC- WT
[    0.001075] total RAM covered: 3583M
[    0.001273] Found optimal setting for mtrr clean up
[    0.001275]  gran_size: 64K chunk_size: 64M num_reg: 4   lose
cover RAM: 0G
[    0.007671] check: Scanning 1 areas for low memory corruption
[    0.007677] Using GB pages for direct mapping
[    0.007943] No NUMA configuration found
[    0.007944] Faking a node at [mem
0x0000000000000000-0x00000000048fffff]
[    0.007954] NODE_DATA(0) allocated [mem 0x048d5000-0x048fffff]
[    0.008371] Zone ranges:
[    0.008372]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.008374]   DMA32    [mem 0x0000000001000000-0x00000000048fffff]
[    0.008375]   Normal   empty
[    0.008376]   Device   empty
[    0.008377] Movable zone start for each node
[    0.008381] Early memory node ranges
[    0.008383]   node   0: [mem 0x0000000000001000-0x000000000009ffff]
[    0.008384]   node   0: [mem 0x0000000000200000-0x00000000048fffff]
[    0.008546] Zeroed struct page in unavailable ranges: 14433 pages
[    0.008547] Initmem setup node 0 [mem
0x0000000000001000-0x00000000048fffff]
[    0.008887] SFI: Simple Firmware Interface v0.81
http://simplefirmware.org
[    0.008957] IOAPIC[0]: apic_id 0, version 33, address 0xfec00000,
GSI 0-23
[    0.008959] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.008968] PM: hibernation: Registered nosave memory: [mem
0x00000000-0x00000fff]
[    0.008969] PM: hibernation: Registered nosave memory: [mem
0x00002000-0x00002fff]
[    0.008971] PM: hibernation: Registered nosave memory: [mem
0x000a0000-0x000fffff]
[    0.008972] PM: hibernation: Registered nosave memory: [mem
0x00100000-0x00100fff]
[    0.008973] PM: hibernation: Registered nosave memory: [mem
0x00101000-0x001fffff]
[    0.008975] [mem 0x04900000-0xffffffff] available for PCI devices
[    0.008976] Booting paravirtualized kernel on bare hardware
[    0.008980] clocksource: refined-jiffies: mask: 0xffffffff
max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.008985] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:2
nr_cpu_ids:2 nr_node_ids:1
[    0.009399] percpu: Embedded 56 pages/cpu s192512 r8192 d28672
u1048576
[    0.009426] Built 1 zonelists, mobility grouping on.  Total pages:
18026
[    0.009427] Policy zone: DMA32
[    0.009428] Kernel command line: console=ttyS0,9600
[    0.009468] Dentry cache hash table entries: 16384 (order: 5,
131072 bytes, linear)
[    0.009478] Inode-cache hash table entries: 8192 (order: 4, 65536
bytes, linear)
[    0.009498] mem auto-init: stack:off, heap alloc:on, heap free:off
[    0.009620] Memory: 37524K/73340K available (14339K kernel code,
2386K rwdata, 5020K rodata, 2480K init, 5240K bss, 35816K reserved, 0K
[    0.009629] random: get_random_u64 called from
__kmem_cache_create+0x42/0x540 with crng_init=0
[    0.009763] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2,
Nodes=1
[    0.009779] ftrace: allocating 43599 entries in 171 pages
[    0.028212] ftrace: allocated 171 pages with 5 groups
[    0.028311] rcu: Hierarchical RCU implementation.
[    0.028312] rcu: RCU restricting CPUs from NR_CPUS=8192 to
nr_cpu_ids=2.
[    0.028314] Tasks RCU enabled.
[    0.028315] rcu: RCU calculated value of scheduler-enlistment
delay is 100 jiffies.
[    0.028316] rcu: Adjusting geometry for rcu_fanout_leaf=16,
nr_cpu_ids=2
[    0.031885] NR_IRQS: 524544, nr_irqs: 424, preallocated irqs: 0
[    0.032007] random: crng done (trusting CPU's manufacturer)
[    0.032029] Console: colour dummy device 80x25
[    0.032031] Enabling UART0 (port 0x3f8)
[    5.771306] printk: console [ttyS0] enabled
[    5.820808] APIC: Switch to symmetric I/O mode setup
[    5.879554] Switched APIC routing to physical flat.
FATAL: Setting invalid LVT delivery mode (reg 35, value 00000700)
FATAL: Unhandled APIC access, offset 848, is_write: 1


Smells like a regression in that branch for non-root Linux. Is that
562b04e51bb5e2f04bf175383080333237067c63? Can you share you kernel
config?

RIP: 0xffffffffab671f72 RSP: 0xffffffffacc03e40 FLAGS: 246
RAX: 0xffffffffab671f70 RBX: 0x0000000000000180 RCX: 0x0000000000000000
RDX: 0x0000000000000000 RSI: 0x0000000000000700 RDI: 0x0000000000000350
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x000000000260a000 CR4: 0x00000000000406b0
EFER: 0x0000000000001d01
Parking CPU 6 (Cell: "linux-x86-demo")


Furthermore, if i try to attach an initrd.image
(generated according to
http://www.aclevername.com/articles/linux-xilinx-tutorial/minimalist-initramfs.html)

to the
command via -i, the following error occurs:


File "/usr/local/libexec/jailhouse/jailhouse-cell-linux", line 744,
in <module>
cell.load(args.initrd.read(), arch.ramdisk_adress())
IOError: [Errno 22] Invalid argument


Possibly too large. Check if the one from jailhouse-images works (it's
small). Also check if that kernel works for you (older baseline).


Now my question are:

   1.   How to configure the non-root linux cell correctly?
   2.   How to boot up a minimal initrd?
   3.   How to compleletey boot a minimal linux distribution?

Please kindly find the attached configuration files (.config,
rootcell.c, linux-x86-cell.c).
Any help would be highly appreciated.


First reference point is always
https://github.com/siemens/jailhouse-images, and then we can look for
the delta that breaks things.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux


--
You received this message because you are subscribed to the Google Groups 
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jailhouse-dev/73b73089-a90c-f26f-bc2f-ce716bb1fafb%40st.oth-regensburg.de.

Re: Need help to run Linux in non-root cell

Reply via email to