Jan & Ralf: I have a little good news, I have successfully gotten the non-root linux to display some startup output based on your suggestions. Also, there must have been some small difference between the stock linux-x86-demo cell config and the one I tweaked. I went back to the original and made some minor changes and it started displaying the serial output now for the non-root node.
I ended up making the root cell the primary user of the ttyS0 device, and just having the non-root linux output to the hypervisor virtual console (console=jailhouse). In my setup, there appears to be some process still attempting to use the ttyS0 randomly on the root linux, and that causes the hypervisor to halt it if I don't leave it in the PIO array. Not sure who is doing it. This is where i'm at now. The non-root linux keeps running into points where it tries to access PIO and gets parked: 1. First crash: [ 0.424925]kworker/u6:0 (27) used greated stack depth: 14656 bytes left [ 0.425807]futex has table entries: 1024 (order: 4, 65536 bytes) "FATAL: Invalid PIO write, port: 70: size 1" ... Parking CPU 3 (Cell: "linux-x86-demo") According to /proc/ioports, 70 has to do with the Realtime clock (rtc0). I was able to temporarily add the 70 mapping to the non-root linux, but should it be there at all? I tried to disable RTC support in the guest kernel config. 2. On the next attempt, I then got further with a crash trying to initialize Ser device ttyS0: "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled" "FATAL: Invalid PIO read, port: 3f9: size 1" ... Parking CPU 3 (Cell: "linux-x86-demo") I can get around that if I allow the non-root access in its pio table, but then I have a problem above with the root linux randomly trying to use it. 3. Now with the temporary adjustments to the non-root pio table above I get here: "[ 0.972399]clocksource:Switched to clocksource tsc" "FATAL: Invalid PIO read, port: 87: size 1" ... Parking CPU 3 (Cell: "linux-x86-demo") According to proc/ioports this one has to do with "dma page request". It isn't mapped in my root linux PIO or non-root linux array. Aside from the serial conflict, it seems like these should remain controlled by the root linux or hypervisor. Do you use a particular bare minimum x86 kernel config on your machine for the guest to get around these? I know Jan mentioned the jailhouse-images project might have something, but I couldn't find it digging around quick. Thanks to you both for your responses and patience. I got the root linux up and going on my own, but the non-root guest is proving to be more difficult. Wayne On Tue, Jun 11, 2019 at 11:31 AM Ralf Ramsauer < [email protected]> wrote: > Hi, > > On 6/10/19 7:45 PM, Wayne wrote: > > Thank you for your responses Ralf and Jan. > > > > Yes, I have successfully gotten both the apic-demo and tiny-demo cells > > to work on my system before attempting to load the non-root linux. I > > can see serial output from them. > > > > However, if I try to share the UART between root and inmate for them I > > run into a crash on the hypervisor if the root linux attempts to write > > to the UART (after creating/starting a demo). Does the root linux lose > > access once an inmate is created/started? Or am I missing something > > from my System config or tiny-demo config that allows them to share? I > > attached the crash in hypervisor_output.txt. > > > > Also, I noticed the tiny-demo output on the UART gets mirrored on the > > Yes, according to your config, JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE is > set. This means that the guest will use the virtual console. The virtual > console uses the hypervisor output which depends whatever is set in your > system configuration. In your case the hypervisor uses 3f8. > > And additionally, the inmate also a valid .console set. Again, 3f8. This > means, it will use the console *and* the hypervisor debug hypercall. In > your case, both outputs are routed to the same device which explains the > mirroring. > > > Jailhouse virtual console. Does that mean that its writing to the > > serial through the hypervisor and not directly accessing ttyS0 itself? > > Exactly. > > > If so, is there a way to create the cell to write ttyS0 directly as a > > test? To see if it exhibits same behavior as the non-root linux. > > Sure, simply align the .address field of .console, and allow access to > that port via .pio_bitmap. > > > > > FYI, I create the tiny-demo like this: > > > > jailhouse cell create configs/x86/tiny-demo.cell > > jailhouse cell load tiny-demo inmates/demos/x86/tiny-demo.cell > > jailhouse cell start tiny-demo > > > > Going back to my original non-root linux question, I don't see any > > output when connected over ttyS0. Its address is 0x3f8 ("serial 1") in > > the PIO, and that seems to work fine for the other simple demos, minus > > the hypervisor crash issue above when sharing. I know there is the > > other "serial 2" port but I haven't tried to use that one. > > I see that port on our Dell server as well. It's present, but I don't > know where it's connected to. > > Let me just summarise your issue: > > You only have one serial line available, right? From the root-cell's > POV, it's ttyS0 on 3f8. > > I would propose to share ttyS0 between the hypervisor and non-root > Linux. This means: > > - Don't let the root-cell use ttyS0. Remove any console=ttyS0 > parameters from you commandline. Ensure that noone else accesses > ttyS0 (e.g., getty@ttyS0 or other friends) > > - Set 3f8 as debug_console in your master.c (seems you already have) > > - Allow 3f8 access in the linux-demo.c (already set in the default > linux-demo > > - Disallow 2f8 access in linux-demo.c: > - [ 0x2f8/8 ... 0x2ff/8] = 0, /* serial2 */ > + [ 0x2f8/8 ... 0x2ff/8] = -1, /* serial2 */ > > I don't know how Linux enumerates serial devices, but this ensures > that Linux won't see the unconnected serial line and map ttyS0 to > 3f8 > > > > > non root linux launched with: > > /tools/jailhouse cell linux configs/x86/linux-x86.cell > > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200" > > Then, this should actually work... > > And if not, then console=jailhouse0 should work. > > > > > I can try the 4.19 siemens kernel and "next" branch for jailhouse you > > suggested. Do you think that combination will resolve the missing > > serial for the non-root linux? Otherwise, did my kernel config, system > > config, and non-root linux cell configs look OK? > > I didn't look at it, but we should at least - even if some flags are > still incorrect - see the "Uncompressing kernel..." thing. > > Wait -- one thing you could try: Deactivate CONFIG_EFI and especially > CONFIG_EFI_STUBS. I'm not sure, but it could be that EFI_STUBS change > the header format that our bootloader patches. > > > > > Thanks again for your help. > > No problem. But it's not yet working. ;-) > > Ralf > > > > > > > On Fri, Jun 7, 2019 at 5:30 PM Ralf Ramsauer > > <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Wayne, > > > > On 6/7/19 10:04 PM, Wayne wrote: > > > Hi Ralf, > > > > > > Thank you for your responses. I have attached the requested config > > > files. I am now just trying to use the same bzImage for the root > and > > > the non-root linux node. > > > > One question that you didn't answer: do apic-demo or tiny-demo work? > > They need to to have correct .console parameter set, so ensure that > > parameters are set if you run those demos in your linux-demo cell. > Just > > compare it to tiny-demo.c or apic-demo.c > > > > > > > > Yes, I am using vanilla 4.16 because the documentation indicated > > that it > > > is the first mainline release to officially support Jailhouse Guest > > > images. Do you suggest that I use something else? > > > > As Jan already wrote: Try to use Siemens' 4.19 branch. > > > > > > > > Right now i'm just trying to get some serial output from the > non-root > > > Linux. Hopefully, I have that configured correctly. Ideally, I > would > > > like to share the same serial console for the Root node and > non-root > > > node eventually if possible. I have been OK with using the virtual > > > console (jailhouse console -f) for the Hypervisor output. > > > > In addition to Jan's comment: > > > > Currently you only have access to the jailhouse console via > 'jailhouse > > console' from your root cell. Imagine your root cell crashes for some > > reason. You will never see the fault reason, which makes things hard > to > > debug. > > > > I would suggest to configure the hypervisor to use the serial > console. > > You can share the device with the non-root Linux, that's no problem. > > > > In this case, the non-root kernel's output will always be printed to > the > > serial console. Directly, if you choose console=ttySx, and indirectly > > via the hypervisor if you choose console=jailhouse. > > > > BTW: According to your config, your system is a PowerEdge, and the > > non-root cell gets both, 0x2f8 and 0x3f8. Are you sure that ttyS1 is > the > > correct console that you are connected to? > > > > Just mentioning this as we have a similar machine here, and, afair, > both > > platform serial device are 'usable', but one ends in the nirvana > while > > it is accessible. > > > > Ah, and one last thing: try to switch to the current next branch for > > jailhouse. Jan just integrated a few patches from me that might also > be > > relevant for your machine. > > > > HTH > > Ralf > > > > > > > > Wayne > > > > > > On Fri, Jun 7, 2019 at 9:10 AM Ralf Ramsauer > > > <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] > > <mailto:[email protected]>>> wrote: > > > > > > Hi, > > > > > > On 6/7/19 2:28 PM, Wayne wrote: > > > > Hello, > > > > > > > > I am new to Linux development and Jailhouse. I have > > successfully > > > booted > > > > the Jailhouse Hypervisor and root cell on a bare metal X86 > Linux > > > system > > > > (No QEMU). I am now trying to load a non-root Linux cell > > and I have a > > > > few questions. Jailhouse accepts and starts my non-root > > linux cell > > > > configuration and I see it as "running" through the > > "jailhouse cell > > > > list" command. However, I don't see any serial output from > the > > > > "non-root linux" cell booting up. I’m not sure what the > > non-root node > > > > is doing at this point. > > > > > > Ok, first of all, can you see any output from Jailhouse's demo > > inmates > > > (e.g., tiny-demo or apic-demo)? > > > > > > > > > > > My root node is a 4.16 kernel configured this > > way: > > > > 1. CONFIG_JAILHOUSE_GUEST is > > not set > > > > 2. CONFIG_SERIO=y > > > > 3. > > CONFIG_SERIAL_8250_RUNTIME_UARTS=4 > > > > > > > > My non-root node is a 4.16 kernel configured > > this way: > > > > 1. CONFIG_JAILHOUSE_GUEST=y > > > > 2. CONFIG_SERIO=m (can't seem > > to disable > > > > completely in my config for 4.16) > > > > 3. > > CONFIG_SERIAL_8250_RUNTIME_UARTS=1 > > > > > > Could you please provide a full .config? What branch are you > > exactly > > > using? Vanilla 4.16? > > > > > > Please attach your system config and your non-root linux > > config as well. > > > > > > You may also want to try https://github.com/siemens/linux . > See > > > jailhouse-enabling branches for some releases. > > > > > > > > > > > In general, do the kernel config settings have to match > between > > > the root > > > > node and non-linux or is the above fine? > > > > > > No, they do not have to be the same, but they can. Still, an > > analysis > > > requires your .config. > > > > > > > > > > > The vmlinux-4.1.16-Guest bzImage is approx 7MB, and the > > inmate node is > > > > allocated ~75MB of RAM. > > > > > > > > I have a single UART, so I have configured the root cell > > system config > > > > to output to the virtual hypervisor console: > > > > > > Wait... You configured your root-cell to use the hypervisor > debug > > > console? How? It's only available once the hypervisor is > > enabled. How > > > should this work? > > > > > > But let's have a look at your configuration first. > > > > > > So you want to: > > > - Have the UART (0x3f8) available in the non-root cell > > > - Use 0x3f8 as hypervisor debug console as well > > > - No console for root-cell > > > > > > Did I get this right? > > > > > > Ralf > > > > > > > > > > > .flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE, > > > > > > > > .debug_console = { > > > > .type = JAILHOUSE_CON_TYPE_NONE, > > > > }, > > > > > > > > and I have configured the non-root linux cell to output to > > the UART: > > > > > > > > (Added serial 0x3f8 to pio bitmap for non-root linux) and > > started the > > > > node with this: > > > > > > > > ./tools/jailhouse cell linux configs/x86/linux-x86.cell > > > > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200" > > > > (Note I also tried "console=jailhouse" in the command above, > > and that > > > > produces the same result) > > > > > > > > I then see the following on my hypervisor console > > (./tools/jailhouse > > > > console -f): > > > > > > > > Created cell "linux-x86-demo" > > > > ... > > > > Cell "linux-x86-demo" can be loaded > > > > Started cell "linux-x86-demo" > > > > > > > > After a little while I do get a parked CPU error on the root > > node, > > > looks > > > > like its trying to do something with the UART as well: > > > > FATAL: Invalid PIO read, port: 3fe size: 1 > > > > > > > > I would expect something to pop out on the UART from the > > non-root > > > linux > > > > node first. Note that root node has serial 0x3f8 disabled > > in its pio > > > > bitmap. > > > > > > > > I verifed that the UART is functioning by allowing the > > hypervisor to > > > > print to it and also performed an echo test over ttyS0. > > > > > > > > I have tried several configurations of kernel.....including > your > > > current > > > > "queues/jailhouse" branch head kernel for the non-root node, > > along > > > with > > > > the kernel config for 4.7 posted in this thread below (but I > > get same > > > > result as above when I start it, no kernel output): > > > > > > > > > > > > > " > https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ > ". > > > > > > > > Any information you can provide to me will be helpful. I'm > > not sure > > > > what might be going wrong here. > > > > > > > > Thanks, > > > > Wayne > > > > > > > > -- > > > > You received this message because you are subscribed to the > > Google > > > > Groups "Jailhouse" group. > > > > To unsubscribe from this group and stop receiving emails > > from it, send > > > > an email to [email protected] > > <mailto:jailhouse-dev%[email protected]> > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]>> > > > > <mailto:[email protected] > > <mailto:jailhouse-dev%[email protected]> > > > <mailto:jailhouse-dev%[email protected] > > <mailto:jailhouse-dev%[email protected]>>>. > > > > To view this discussion on the web visit > > > > > > > > > > https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com > > > > > > > > > < > https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com?utm_medium=email&utm_source=footer > >. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc0FZSWvVZZyV%2B4_F0TO7dHPN70Mn3LUCmq_3M2fnGN4ig%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
