Hi Michele,
On 16/01/2024 20:44, Michele Pescapè wrote:
Hi,
I'm trying to give network access to the non root cell, however it seems
that I'm not able to pass the right PCI device. It always defaults to
taking away the 00:00.0 pci device from the root cell causing a crash
for the root cell.
In both the root and non root cell's configs I added the shared memory
regions using the macro JAILHOUSE_SHMEM_NET_REGIONS(0x60000000, 0) for
the root and using dev_id=1 for the non root. The resulting regions do
not overlap other memory regions, config check does not return any errors.
I also added the following PCI device in both cells, where
.shmem_regions_start is assigned to the correct index for the first
shared memory region created using the aforementioned macro.
/* IVSHMEM: 00:05.0 */
{
.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
.domain = 0x0,
.bdf = 0x28,
[…]
.bar_mask = JAILHOUSE_IVSHMEM_BAR_MASK_MSIX,
.num_msix_vectors = 2,
.shmem_regions_start = 0,
.shmem_dev_id = 0, (1 in the non root cell)
.shmem_peers = 2,
.shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
},
I tried using different values for the .bdf, none of which were already
used by other devices but none worked. The root cell always sees the new
device correctly when starting the hypervisor, I can also see it with
lspci and ifconfig, however once I issue the cell linux command to start
the non root cell, I get the following output followed by a streak of
errors from the hypervisor, forcing a manual reset.
Removing PCI device 00:00.0 from cell "RootCell"
Adding PCI device 00:00.0 to cell "linux-1"
.bdf = 0x28 doesn't correspond with 00:00.0. Would you please test your
current configs against jailhouse-config-check, and in case of no
errors, would you please attach the current versions again?
Thanks,
Ralf
FATAL: unable to get MMIO instruction
FATAL: Invalid MMIO/RAM write, addr: 0x0000000106d8c020 size: 0
RIP: 0xffffffffa910ff4c RSP: 0xffffab85400f7de0 FLAGS: 93
RAX: 0x0000000000000000 RBX: 0x000000000002f170 RCX: 0x0000000000000040
RDX: 0x0000000000000414 RSI: 0xffff8b6000955400 RDI: 0x0000000000000001
CS: 10 BASE: 0x0000000000000000 AR-BYTES: 29b EFER.LMA 1
CR0: 0x0000000080050033 CR3: 0x0000000106d8c000 CR4: 0x00000000003506a0
EFER: 0x0000000000001d01
Parking CPU 4 (Cell: "RootCell")
(etc..)
I'm not sure what I'm missing here, CONFIG_IVSHMEM_NET is enabled in
both the root and non root kernel config, I also enabled
CONFIG_PCI_HOST_GENERIC in the root linux as suggested in one of the
conversations on this site.
Thanks,
Michele
Il giorno domenica 14 gennaio 2024 alle 15:49:40 UTC+1 Ralf Ramsauer ha
scritto:
Hi Michele,
On 14/01/2024 02:18, Michele Pescapè wrote:
> Hi,
>
> There were PIO writes to ports 4e,4f,2e and 2f, each of size 1
and each
> one on subsequent restarts. I'm not sure how to figure out what they
> belong to.
Easy and straight forward: Read the jailhouse crash dump.
Look at the program counter, and maybe the return address. Disassemble
your kernel (objdump -d vmlinux) and lookup that address. Now you know
the name of the routine in the kernel that caused the crash.
Don't simply whitelist ports. Only whitelist them, if you're sure what
you're doing.
>
> CONFIG_ISA_DMA_API is already disabled.
>
> I missed the irqchip for the uart. However as of right now I just
copied
> the whole fragment from the root configuration which means I
could be
> taking away other interrupts from the root cell, I still have to
figure
> out how to tune that
>
> /* IOAPIC 13, GSI base 0 */
> {
> .address = 0xfec00000,
> .id = 0xa0,
> .pin_bitmap = {
> 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
With this, you take away ALL interrupts from the root cell, which you
don't want to do, you only want to take away the interrupt that
corresponds to ttyS0 (i.e. Port 0x3f8).
See here: https://en.wikipedia.org/wiki/COM_(hardware_interface)
<https://en.wikipedia.org/wiki/COM_(hardware_interface)>
COM1: I/O port 0x3F8, IRQ 4
So you want to only assign IRQ 4 to your non-root cell. See
linux-x86-demo.c to see how that works.
> },
> },
>
> Still, now I can finally login into the non root linux.
Excellent!
Ralf
>
> Thanks,
> Michele
>
> Il giorno sabato 13 gennaio 2024 alle 21:09:28 UTC+1 Ralf
Ramsauer ha
> scritto:
>
> Hi Michele,
>
> On 13/01/2024 18:08, Michele Pescapè wrote:
> > Hi,
> >
> > The problem was that the ram regions in the non root configuration
> > weren't detected as such because of the missing JAILHOUSE_MEM_DMA
> flag.
>
> Yikes, makes sense.
>
> > After adding that, I also had to add two pio_regions because
the non
> > root linux was crashing because of two ports which aren't
present in
> > /proc/ioports.
>
> Uhm - which PIO ports? Don't simply assign PIO ports. There must
be a
> reason for them. What ports did crash?
>
> Did you disable(!) CONFIG_ISA_DMA_API? Please disable it. Therefore,
> you
> have to activate CONFIG_EXPERT first.
>
> I *bet* it was i8237…
>
> > I also had to add mce=off to the command line because i had an
> unhandled
>
> or disable CONFIG_X86_MCE.
>
> > MSR error, I'll have to disable that in the kernel config.
> > At this point the non root linux seems to start, I see the boot
> log. No
>
> Excellent!
>
> > way of interacting with it as there is not a login prompt or
> anything, I
> > think I need to ssh to the cell at this point, right?
>
> Was the initramdisk loaded correctly?
>
> Did you assign - in your non-root cell config - the irqchip and the
> corresponding interrupts of the uart?
>
> > That means I'll now need to work on those ivshmem net devices.
>
> If you need ivshmem, then this would be the next step.
>
> Ralf
>
> >
> > Thanks,
> > Michele
> >
> > Il giorno sabato 13 gennaio 2024 alle 15:13:12 UTC+1 Ralf
> Ramsauer ha
> > scritto:
> >
> > Hi,
> >
> > On 13/01/2024 12:28, Michele Pescapè wrote:
> > > Hi,
> > >
> > > You are right, I got confused about those addresses, my bad. At
> > least
> > > now I know that config is the correct one and I don't have to
> tinker
> > > with it.
> > > I'm back to a kernel panic from the inmate when booting the
cell.
> > It's
> > > similar to the one I had earlier, not sure yet of what the
> > problem may be.
> >
> > Great, we're a step further.
> >
> > >
> > > Created cell "linux-2"
> > > Page pool usage after cell creation: mem 406/32211, remap
> > 16392/131072
> > > Cell "linux-2" can be loaded
> > > CPU 9 received SIPI, vector 100
> > > Started cell "linux-2"
> > > CPU 8 received SIPI, vector 100
> > > No EFI environment detected.
> > > early console in extract_kernel
> > > input_data: 0x000000000275c308
> > > input_len: 0x00000000008b0981
> > > output: 0x0000000001000000
> > > output_len: 0x0000000001fccb30
> > > kernel_total_size: 0x0000000001e28000
> > > needed_size: 0x0000000002000000
> > > trampoline_32bit: 0x000000000009d000
> > >
> > > Decompressing Linux... Parsing ELF... done.
> > > Booting the kernel.
> > > [ 0.000000] Linux version 6.2.0-rc3 (root@mp-LINUX-DESKTOP)
> > > (x86_64-buildroot -linux-gnu-gcc.br_real (Buildroot
2023.11)
> > > 12.3.0, GNU ld (GNU Binutils) 2.40) # 2 SMP
PREEMPT_DYNAMIC
> > Fri
> > > Jan 12 17:36:57 CET 2024
> > > [ 0.000000] Command line: earlyprintk=ttyS0,115200
> > > [ 0.000000] KERNEL supported cpus:
> > > [ 0.000000] Intel GenuineIntel
> > > [ 0.000000] AMD AuthenticAMD
> > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87
> > floating
> > > point regi sters'
> > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE
> > registers'
> > > [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX
> > registers'
> > > [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]:
> > 256
> > > [ 0.000000] x86/fpu: Enabled xstate features 0x7, context
size
> > is 832
> > > bytes, using 'compacted' format.
> > > [ 0.000000] signal: max sigframe size: 1360
> > > [ 0.000000] BIOS-provided physical RAM map:
> > > [ 0.000000] BIOS-e801: [mem
> > 0x0000000000000000-0x000000000009efff] usable
> >
> > Okay, here should be all memory regions listed. My non-root Linux
> in my
> > Qemu VM, for example, shows here:
> >
> > [ 0.000000] BIOS-provided physical RAM map:
> > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x00000000000fffff]
> > usable
> > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000000100fff]
> > reserved
> > [ 0.000000] BIOS-e820: [mem 0x0000000000200000-0x00000000048fffff]
> > usable
> >
> > Are you absolutely sure, that you have no further modifications in
> > Jailhouse or the Linux loader?
> >
> > Actually, in your case, there should be e820 as well (instead of
> e801).
> > Go to the Linux kernel sources, have a look at
> > arch/x86/kernel/e820.c:1279
> >
> > and Jailhouse's jailhouse-cell-linux:638.
> >
> > jailhouse-cell-linux fills information of all memory regions into
> the
> > zero page. Would you please instrument code (Linux/Jailhouse)
to see
> > where those regions are not parsed?
> >
> > > [ 0.000000] printk: bootconsole [earlyser0] enabled
> > > [ 0.000000] NX (Execute Disable) protection: active
> > > [ 0.000000] extended physical RAM map:
> > > [ 0.000000] reserve setup_data: [mem
> > > 0x0000000000000000-0x0000000000001fff] u sable
> > > [ 0.000000] reserve setup_data: [mem
> > > 0x0000000000002000-0x000000000000212b] u sable
> > > [ 0.000000] reserve setup_data: [mem
> > > 0x000000000000212c-0x000000000009efff] u sable
> > > [ 0.000000] DMI not present or invalid.
> > > [ 0.000000] Hypervisor detected: Jailhouse
> >
> > Just guessing loud: Hmm, you have guest support enabled, that's
> not the
> > issue.
> >
> > > [ 0.000000] tsc: Detected 3393.624 MHz processor
> > > [ 0.000017] .text .data .bss are not marked as E820_TYPE_RAM!
> >
> > Here's the next error that makes me curious, why you system falls
> back
> > to E801...
> >
> > > [ 0.005745] last_pfn = 0x2e28 max_arch_pfn = 0x400000000
> > > [ 0.011025] x86/PAT: PAT support disabled because
> > CONFIG_X86_PAT is
> > > disabled in the kernel.
> >
> > Please enable CONFIG_X86_PAT and MTRR in your kernel.
> >
> > > [ 0.019362] x86/PAT: Configuration [0-7]: WB WT UC- UC WB
> > WT UC- UC
> > > [ 0.034867] Using GB pages for direct mapping
> > > [ 0.039193] Kernel panic - not syncing: alloc_low_pages: can
> not
> > > alloc memory
> >
> > Yeah, that's the logical aftereffect after the errors above.
> >
> > Thanks,
> > Ralf
> >
> > > [ 0.046183] CPU: 0 PID: 0 Comm: swapper Not tainted
> 6.2.0-rc3 #2
> > > [ 0.052176] Call Trace:
> > > [ 0.054606] <TASK>
> > > [ 0.056691] ? dump_stack_lvl+0x33/0x4e
> > > [ 0.060510] ? panic+0x157/0x303
> > > [ 0.063723] ? sprintf+0x56/0x80
> > > [ 0.066936] ? alloc_low_pages+0x70/0x1a0
> > > [ 0.070930] ? phys_pmd_init+0x1fc/0x2eb
> > > [ 0.074839] ? phys_pud_init+0x116/0x2d3
> > > [ 0.078744] ? __kernel_physical_mapping_init+0x11a/0x290
> > > [ 0.084128] ? init_memory_mapping+0x25e/0x3b0
> > > [ 0.088558] ? init_range_memory_mapping+0xe7/0x145
> > > [ 0.093417] ? init_mem_mapping+0x242/0x298
> > > [ 0.097585] ? setup_arch+0x74e/0xcbd
> > > [ 0.101231] ? start_kernel+0x66/0x8b7
> > > [ 0.104965] ? load_ucode_bsp+0x43/0x11b
> > > [ 0.108873] ? secondary_startup_64_no_verify+0xe0/0xeb
> > > [ 0.114085] </TASK>
> > > [ 0.116255] ---[ end Kernel panic - not syncing:
> > alloc_low_pages: can
> > > not all oc memory ]---
> > >
> > >
> > > Thank you for your continuous support,
> > > Michele
> > >
> > > Il giorno sabato 13 gennaio 2024 alle 00:05:43 UTC+1 Ralf
> > Ramsauer ha
> > > scritto:
> > >
> > > Hi Michele,
> > >
> > > On 12/01/2024 14:07, Michele Pescapè wrote:
> > > > jailhouse cell load linux-2 linux-loader.bin -a 0x0
> > > > ../buildroot-2023.11/output/images/bzImage -a 0xffbe00
> parameters
> > > -a 0x1000
> > > > jailhouse cell start linux-2
> > > >
> > > > I take it the kernel is loaded at 0xffbe00 which is right
at the
> > > edge of
> > > > the low ram region. In fact after issuing the cell load
command
> > and
> > > > adjusting the path for the loader I get JAILHOUSE_CELL_LOAD:
> > Invalid
> > > > argument.
> > >
> > > Just tested cell-linux in a qemu machine, there it works, with
> > pretty
> > > similar addresses, which got me confused.
> > >
> > > After double-checking it: 0xffb.e00 is *not* at the edge of low
> mem:
> > >
> > > Low mem is 0x000.000 -- 0x0ff.fff
> > > Comm region is 0x100.000 -- 0x100.fff
> > >
> > > 0xffb.e00 is (obviously) way above.
> > >
> > > Please try to set your high mem's .virt_start to 0x200000. Then,
> > > 0xffbe00 is offsetted ~16MB inside your highmem, and it should
> work!
> > >
> > > IOW:
> > >
> > > /* high RAM */
> > > {
> > > .phys_start = 0x42300000,
> > > .virt_start = 0x200000,
> > > .size = 0x11000000,
> > > .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> > > JAILHOUSE_MEM_EXECUTE |
> > > JAILHOUSE_MEM_LOADABLE,
> > > },
> > >
> > > Thanks
> > > Ralf
> > >
> > > --
> > > You received this message because you are subscribed to the
Google
> > > Groups "Jailhouse" group.
> > > To unsubscribe from this group and stop receiving emails from
it,
> > send
> > > an email to [email protected]
> > > <mailto:[email protected]>.
> > > To view this discussion on the web visit
> > >
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com>>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6712361a-eaef-49cc-8a72-79da2c434169n%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Jailhouse" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to [email protected]
> > <mailto:[email protected]>.
> > To view this discussion on the web visit
> >
>
https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com>>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer>
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/860c1e75-c28f-4157-9212-a3d87ad25b27n%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jailhouse" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com>
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/jailhouse-dev/6146e208-1e8c-4335-abaf-9a2be8c85be5n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
--
You received this message because you are subscribed to the Google
Groups "Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/6fb04e63-2f4a-4d5b-9139-c11e9a9fc461n%40googlegroups.com <https://groups.google.com/d/msgid/jailhouse-dev/6fb04e63-2f4a-4d5b-9139-c11e9a9fc461n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/5fc9ef1f-e7ca-496f-aab5-c085c4586326%40oth-regensburg.de.
