Jan/Ralf, I was able to get past issues 1 and 3 above by using Jan's kernel config from this thread:
https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ ". The DMA issue was resolved by disabling kernel option CONFIG_ISA_DMA_API as stated in another thread, and reflected in the config file above. So now I think i've almost got the non-root linux booting. I'm stuck at the point where its trying to mount a filesystem and since i'm not providing an initramfs it panics. Similar to the output below: [ 1.080178] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6 [ 1.087662] Please append a correct "root=" boot option; here are the available partitions: [ 1.096013] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) I have just a couple more questions: 1. In general, is it true that an inmate will takeover control from the root of any resource in the following arrays, or are there some cases where they get "shared" between the root and inmate? A. ".mem_regions" -> Shared? B. ".irqchips" -> Shared? C. ".pio_bitmap" -> Inmate takes control of these if matching between inmate and root D. ".pci_devices" -> Inmate takes control? E. ".pci_caps" -> Inmate takes control? 2. Along the same lines, going back to my ttyS0 issue mentioned above while trying to share it: If I allocate the following to the linux non root cell (and set to -1 in the root) .pio_bitmap = { [0x3f8/8 ... 0x3ff/8] = 0x00 /* serial 2*/ } Then on the root cell I randomly get a PIO read fault on "0x3FE" when the non-root is booting, i'm not sure what process is causing this. if I leave out the pio mapping in the non-root linux, and alolothat block in the root cell instead then the non-root linux gets a PIO read fault on "0x3F9" while booting. Ralf mentioned that Linux would not try enumerate the device if its set to -1 ( [0x3f8/8 ... 0x3ff/8] = -1) , but the non-root seems to attempt to do something with it anyway: "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled" "FATAL: Invalid PIO read, port: 3f9: size 1" Is it possible to map the same pio block in both the root and non-root with a different mask? Or does the non-root just override it? It seems like the latter is true. Thanks again Jan and Ralf for your help getting to this point. On Tue, Jun 11, 2019 at 2:13 PM Wayne <[email protected]> wrote: > Jan & Ralf: > > I have a little good news, I have successfully gotten the non-root linux > to display some startup output based on your suggestions. Also, there must > have been some small difference between the stock linux-x86-demo cell > config and the one I tweaked. I went back to the original and made some > minor changes and it started displaying the serial output now for the > non-root node. > > I ended up making the root cell the primary user of the ttyS0 device, and > just having the non-root linux output to the hypervisor virtual console > (console=jailhouse). In my setup, there appears to be some process still > attempting to use the ttyS0 randomly on the root linux, and that causes the > hypervisor to halt it if I don't leave it in the PIO array. Not sure who > is doing it. > > > This is where i'm at now. The non-root linux keeps running into points > where it tries to access PIO and gets parked: > > 1. First crash: > > [ 0.424925]kworker/u6:0 (27) used greated stack depth: 14656 bytes left > [ 0.425807]futex has table entries: 1024 (order: 4, 65536 bytes) > "FATAL: Invalid PIO write, port: 70: size 1" > ... > Parking CPU 3 (Cell: "linux-x86-demo") > > According to /proc/ioports, 70 has to do with the Realtime clock (rtc0). > I was able to temporarily add the 70 mapping to the non-root linux, but > should it be there at all? I tried to disable RTC support in the guest > kernel config. > > > 2. On the next attempt, I then got further with a crash trying to > initialize Ser device ttyS0: > > "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled" > "FATAL: Invalid PIO read, port: 3f9: size 1" > ... > Parking CPU 3 (Cell: "linux-x86-demo") > > I can get around that if I allow the non-root access in its pio table, but > then I have a problem above with the root linux randomly trying to use it. > > > 3. Now with the temporary adjustments to the non-root pio table above I > get here: > > "[ 0.972399]clocksource:Switched to clocksource tsc" > "FATAL: Invalid PIO read, port: 87: size 1" > ... > Parking CPU 3 (Cell: "linux-x86-demo") > > According to proc/ioports this one has to do with "dma page request". It > isn't mapped in my root linux PIO or non-root linux array. > > > Aside from the serial conflict, it seems like these should remain > controlled by the root linux or hypervisor. Do you use a particular bare > minimum x86 kernel config on your machine for the guest to get around > these? I know Jan mentioned the jailhouse-images project might have > something, but I couldn't find it digging around quick. > > Thanks to you both for your responses and patience. I got the root linux > up and going on my own, but the non-root guest is proving to be more > difficult. > > Wayne > > > > On Tue, Jun 11, 2019 at 11:31 AM Ralf Ramsauer < > [email protected]> wrote: > >> Hi, >> >> On 6/10/19 7:45 PM, Wayne wrote: >> > Thank you for your responses Ralf and Jan. >> > >> > Yes, I have successfully gotten both the apic-demo and tiny-demo cells >> > to work on my system before attempting to load the non-root linux. I >> > can see serial output from them. >> > >> > However, if I try to share the UART between root and inmate for them I >> > run into a crash on the hypervisor if the root linux attempts to write >> > to the UART (after creating/starting a demo). Does the root linux lose >> > access once an inmate is created/started? Or am I missing something >> > from my System config or tiny-demo config that allows them to share? I >> > attached the crash in hypervisor_output.txt. >> > >> > Also, I noticed the tiny-demo output on the UART gets mirrored on the >> >> Yes, according to your config, JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE is >> set. This means that the guest will use the virtual console. The virtual >> console uses the hypervisor output which depends whatever is set in your >> system configuration. In your case the hypervisor uses 3f8. >> >> And additionally, the inmate also a valid .console set. Again, 3f8. This >> means, it will use the console *and* the hypervisor debug hypercall. In >> your case, both outputs are routed to the same device which explains the >> mirroring. >> >> > Jailhouse virtual console. Does that mean that its writing to the >> > serial through the hypervisor and not directly accessing ttyS0 itself? >> >> Exactly. >> >> > If so, is there a way to create the cell to write ttyS0 directly as a >> > test? To see if it exhibits same behavior as the non-root linux. >> >> Sure, simply align the .address field of .console, and allow access to >> that port via .pio_bitmap. >> >> > >> > FYI, I create the tiny-demo like this: >> > >> > jailhouse cell create configs/x86/tiny-demo.cell >> > jailhouse cell load tiny-demo inmates/demos/x86/tiny-demo.cell >> > jailhouse cell start tiny-demo >> > >> > Going back to my original non-root linux question, I don't see any >> > output when connected over ttyS0. Its address is 0x3f8 ("serial 1") in >> > the PIO, and that seems to work fine for the other simple demos, minus >> > the hypervisor crash issue above when sharing. I know there is the >> > other "serial 2" port but I haven't tried to use that one. >> >> I see that port on our Dell server as well. It's present, but I don't >> know where it's connected to. >> >> Let me just summarise your issue: >> >> You only have one serial line available, right? From the root-cell's >> POV, it's ttyS0 on 3f8. >> >> I would propose to share ttyS0 between the hypervisor and non-root >> Linux. This means: >> >> - Don't let the root-cell use ttyS0. Remove any console=ttyS0 >> parameters from you commandline. Ensure that noone else accesses >> ttyS0 (e.g., getty@ttyS0 or other friends) >> >> - Set 3f8 as debug_console in your master.c (seems you already have) >> >> - Allow 3f8 access in the linux-demo.c (already set in the default >> linux-demo >> >> - Disallow 2f8 access in linux-demo.c: >> - [ 0x2f8/8 ... 0x2ff/8] = 0, /* serial2 */ >> + [ 0x2f8/8 ... 0x2ff/8] = -1, /* serial2 */ >> >> I don't know how Linux enumerates serial devices, but this ensures >> that Linux won't see the unconnected serial line and map ttyS0 to >> 3f8 >> >> > >> > non root linux launched with: >> > /tools/jailhouse cell linux configs/x86/linux-x86.cell >> > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200" >> >> Then, this should actually work... >> >> And if not, then console=jailhouse0 should work. >> >> > >> > I can try the 4.19 siemens kernel and "next" branch for jailhouse you >> > suggested. Do you think that combination will resolve the missing >> > serial for the non-root linux? Otherwise, did my kernel config, system >> > config, and non-root linux cell configs look OK? >> >> I didn't look at it, but we should at least - even if some flags are >> still incorrect - see the "Uncompressing kernel..." thing. >> >> Wait -- one thing you could try: Deactivate CONFIG_EFI and especially >> CONFIG_EFI_STUBS. I'm not sure, but it could be that EFI_STUBS change >> the header format that our bootloader patches. >> >> > >> > Thanks again for your help. >> >> No problem. But it's not yet working. ;-) >> >> Ralf >> >> > >> > >> > On Fri, Jun 7, 2019 at 5:30 PM Ralf Ramsauer >> > <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi Wayne, >> > >> > On 6/7/19 10:04 PM, Wayne wrote: >> > > Hi Ralf, >> > > >> > > Thank you for your responses. I have attached the requested >> config >> > > files. I am now just trying to use the same bzImage for the root >> and >> > > the non-root linux node. >> > >> > One question that you didn't answer: do apic-demo or tiny-demo work? >> > They need to to have correct .console parameter set, so ensure that >> > parameters are set if you run those demos in your linux-demo cell. >> Just >> > compare it to tiny-demo.c or apic-demo.c >> > >> > > >> > > Yes, I am using vanilla 4.16 because the documentation indicated >> > that it >> > > is the first mainline release to officially support Jailhouse >> Guest >> > > images. Do you suggest that I use something else? >> > >> > As Jan already wrote: Try to use Siemens' 4.19 branch. >> > >> > > >> > > Right now i'm just trying to get some serial output from the >> non-root >> > > Linux. Hopefully, I have that configured correctly. Ideally, I >> would >> > > like to share the same serial console for the Root node and >> non-root >> > > node eventually if possible. I have been OK with using the >> virtual >> > > console (jailhouse console -f) for the Hypervisor output. >> > >> > In addition to Jan's comment: >> > >> > Currently you only have access to the jailhouse console via >> 'jailhouse >> > console' from your root cell. Imagine your root cell crashes for >> some >> > reason. You will never see the fault reason, which makes things >> hard to >> > debug. >> > >> > I would suggest to configure the hypervisor to use the serial >> console. >> > You can share the device with the non-root Linux, that's no problem. >> > >> > In this case, the non-root kernel's output will always be printed >> to the >> > serial console. Directly, if you choose console=ttySx, and >> indirectly >> > via the hypervisor if you choose console=jailhouse. >> > >> > BTW: According to your config, your system is a PowerEdge, and the >> > non-root cell gets both, 0x2f8 and 0x3f8. Are you sure that ttyS1 >> is the >> > correct console that you are connected to? >> > >> > Just mentioning this as we have a similar machine here, and, afair, >> both >> > platform serial device are 'usable', but one ends in the nirvana >> while >> > it is accessible. >> > >> > Ah, and one last thing: try to switch to the current next branch for >> > jailhouse. Jan just integrated a few patches from me that might >> also be >> > relevant for your machine. >> > >> > HTH >> > Ralf >> > >> > > >> > > Wayne >> > > >> > > On Fri, Jun 7, 2019 at 9:10 AM Ralf Ramsauer >> > > <[email protected] >> > <mailto:[email protected]> >> > > <mailto:[email protected] >> > <mailto:[email protected]>>> wrote: >> > > >> > > Hi, >> > > >> > > On 6/7/19 2:28 PM, Wayne wrote: >> > > > Hello, >> > > > >> > > > I am new to Linux development and Jailhouse. I have >> > successfully >> > > booted >> > > > the Jailhouse Hypervisor and root cell on a bare metal X86 >> Linux >> > > system >> > > > (No QEMU). I am now trying to load a non-root Linux cell >> > and I have a >> > > > few questions. Jailhouse accepts and starts my non-root >> > linux cell >> > > > configuration and I see it as "running" through the >> > "jailhouse cell >> > > > list" command. However, I don't see any serial output from >> the >> > > > "non-root linux" cell booting up. I’m not sure what the >> > non-root node >> > > > is doing at this point. >> > > >> > > Ok, first of all, can you see any output from Jailhouse's demo >> > inmates >> > > (e.g., tiny-demo or apic-demo)? >> > > >> > > > >> > > > My root node is a 4.16 kernel configured this >> > way: >> > > > 1. CONFIG_JAILHOUSE_GUEST is >> > not set >> > > > 2. CONFIG_SERIO=y >> > > > 3. >> > CONFIG_SERIAL_8250_RUNTIME_UARTS=4 >> > > > >> > > > My non-root node is a 4.16 kernel configured >> > this way: >> > > > 1. CONFIG_JAILHOUSE_GUEST=y >> > > > 2. CONFIG_SERIO=m (can't seem >> > to disable >> > > > completely in my config for 4.16) >> > > > 3. >> > CONFIG_SERIAL_8250_RUNTIME_UARTS=1 >> > > >> > > Could you please provide a full .config? What branch are you >> > exactly >> > > using? Vanilla 4.16? >> > > >> > > Please attach your system config and your non-root linux >> > config as well. >> > > >> > > You may also want to try https://github.com/siemens/linux . >> See >> > > jailhouse-enabling branches for some releases. >> > > >> > > > >> > > > In general, do the kernel config settings have to match >> between >> > > the root >> > > > node and non-linux or is the above fine? >> > > >> > > No, they do not have to be the same, but they can. Still, an >> > analysis >> > > requires your .config. >> > > >> > > > >> > > > The vmlinux-4.1.16-Guest bzImage is approx 7MB, and the >> > inmate node is >> > > > allocated ~75MB of RAM. >> > > > >> > > > I have a single UART, so I have configured the root cell >> > system config >> > > > to output to the virtual hypervisor console: >> > > >> > > Wait... You configured your root-cell to use the hypervisor >> debug >> > > console? How? It's only available once the hypervisor is >> > enabled. How >> > > should this work? >> > > >> > > But let's have a look at your configuration first. >> > > >> > > So you want to: >> > > - Have the UART (0x3f8) available in the non-root cell >> > > - Use 0x3f8 as hypervisor debug console as well >> > > - No console for root-cell >> > > >> > > Did I get this right? >> > > >> > > Ralf >> > > >> > > > >> > > > .flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE, >> > > > >> > > > .debug_console = { >> > > > .type = JAILHOUSE_CON_TYPE_NONE, >> > > > }, >> > > > >> > > > and I have configured the non-root linux cell to output to >> > the UART: >> > > > >> > > > (Added serial 0x3f8 to pio bitmap for non-root linux) and >> > started the >> > > > node with this: >> > > > >> > > > ./tools/jailhouse cell linux configs/x86/linux-x86.cell >> > > > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200" >> > > > (Note I also tried "console=jailhouse" in the command above, >> > and that >> > > > produces the same result) >> > > > >> > > > I then see the following on my hypervisor console >> > (./tools/jailhouse >> > > > console -f): >> > > > >> > > > Created cell "linux-x86-demo" >> > > > ... >> > > > Cell "linux-x86-demo" can be loaded >> > > > Started cell "linux-x86-demo" >> > > > >> > > > After a little while I do get a parked CPU error on the root >> > node, >> > > looks >> > > > like its trying to do something with the UART as well: >> > > > FATAL: Invalid PIO read, port: 3fe size: 1 >> > > > >> > > > I would expect something to pop out on the UART from the >> > non-root >> > > linux >> > > > node first. Note that root node has serial 0x3f8 disabled >> > in its pio >> > > > bitmap. >> > > > >> > > > I verifed that the UART is functioning by allowing the >> > hypervisor to >> > > > print to it and also performed an echo test over ttyS0. >> > > > >> > > > I have tried several configurations of kernel.....including >> your >> > > current >> > > > "queues/jailhouse" branch head kernel for the non-root node, >> > along >> > > with >> > > > the kernel config for 4.7 posted in this thread below (but I >> > get same >> > > > result as above when I start it, no kernel output): >> > > > >> > > > >> > > >> > " >> https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ >> ". >> > > > >> > > > Any information you can provide to me will be helpful. I'm >> > not sure >> > > > what might be going wrong here. >> > > > >> > > > Thanks, >> > > > Wayne >> > > > >> > > > -- >> > > > You received this message because you are subscribed to the >> > Google >> > > > Groups "Jailhouse" group. >> > > > To unsubscribe from this group and stop receiving emails >> > from it, send >> > > > an email to [email protected] >> > <mailto:jailhouse-dev%[email protected]> >> > > <mailto:jailhouse-dev%[email protected] >> > <mailto:jailhouse-dev%[email protected]>> >> > > > <mailto:[email protected] >> > <mailto:jailhouse-dev%[email protected]> >> > > <mailto:jailhouse-dev%[email protected] >> > <mailto:jailhouse-dev%[email protected]>>>. >> > > > To view this discussion on the web visit >> > > > >> > > >> > >> https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com >> > > > >> > > >> > < >> https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com?utm_medium=email&utm_source=footer >> >. >> > > > For more options, visit https://groups.google.com/d/optout. >> > > >> > >> > -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc0O2zSeuLLY3MaeRW7cQrWbq-6Y2BHJg%2Bx_j6nk%3DECa_A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
