Hi,
On 6/11/19 11:05 PM, Wayne wrote:
> Jan/Ralf,
>
> I was able to get past issues 1 and 3 above by using Jan's kernel config
> from this thread:
>
> https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ";.
>
>
> The DMA issue was resolved by disabling kernel option CONFIG_ISA_DMA_API
> as stated in another thread, and reflected in the config file above.
>
> So now I think i've almost got the non-root linux booting. I'm stuck at
> the point where its trying to mount a filesystem and since i'm not
> providing an initramfs it panics.
Perfect!
>
> Similar to the output below:
>
> [ 1.080178] VFS: Cannot open root device "(null)" or unknown-block(0,0):
> error -6
> [ 1.087662] Please append a correct "root=" boot option; here are the
> available partitions:
> [ 1.096013] Kernel panic - not syncing: VFS: Unable to mount root fs on
> unknown-block(0,0)
>
> I have just a couple more questions:
> 1. In general, is it true that an inmate will takeover control from the
> root of any resource in the following arrays, or are there some cases
> where they get "shared" between the root and inmate?
Depends.
> A. ".mem_regions" -> Shared?
May be shared with the root cell (i.e., if the region has the flag
JAILHOUSE_MEM_ROOTSHARED set).
> B. ".irqchips" -> Shared?
Shared with whom? Only one guest runs on a CPU. This guest will have
full access to the irqchip. Some accesses must be intercepted.
> C. ".pio_bitmap" -> Inmate takes control of these if matching
> between inmate and root
Exclusive control.
> D. ".pci_devices" -> Inmate takes control?
Exclusive control.
> E. ".pci_caps" -> Inmate takes control?
Exclusive control.
>
> 2. Along the same lines, going back to my ttyS0 issue mentioned above
> while trying to share it:
>
> If I allocate the following to the linux non root cell (and set
> to -1 in the root)
BTW, there's generally no need to set -1 in the root cell -- it will be
taken away when the non-root cell is created.
> .pio_bitmap = {
> [0x3f8/8 ... 0x3ff/8] = 0x00 /* serial 2*/
> }
>
> Then on the root cell I randomly get a PIO read fault on "0x3FE"
> when the non-root is booting, i'm not sure what process is causing this.
- Are you sure you remove the device from the kernel cmdline?
- Are you sure that there doesn't spawn a tty? Check with systemctl.
- lsof | grep ttyS
- The root-cell might receive the serial device's interrupt, while its
ports are assigned to the non-root cell. Does the crash happen when
you type in a character to the serial line?
In this case, assign the interrupt to the non-root cell. You will need
it in any case there once you have a initrd and want to type in stuff.
So there must be some reason why the crash happens. The hard way is to
add 8250.nr_uarts=0 in your kernel cmdline of the root cell.
>
> if I leave out the pio mapping in the non-root linux, and
> alolothat block in the root cell instead then the non-root linux gets a
> PIO read fault on "0x3F9" while booting.
> Ralf mentioned that Linux would not try enumerate the device if
> its set to -1 ( [0x3f8/8 ... 0x3ff/8] = -1) , but the non-root seems to
> attempt to do something with it anyway:
>
> "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled"
> "FATAL: Invalid PIO read, port: 3f9: size 1"
Hmm. At least that's what I thought. I don't know how enumeration of
platform devices works exactly, but reconsidering this, the behaviour
above makes sense: How would it know that a platform device is not
present, depending of the configuration of the pio_bitmap.
Anyway, 8250 drivers have some cmdline parameters that can be used to
control initialisation (e.g., see the nr_uarts parameters above).
>
> Is it possible to map the same pio block in both the root and
> non-root with a different mask? Or does the non-root just override it?
> It seems like the latter is true.
Exactly. That's not a) possible, and b) does not really make sense.
Ralf
>
> Thanks again Jan and Ralf for your help getting to this point.
>
>
> On Tue, Jun 11, 2019 at 2:13 PM Wayne <[email protected]
> <mailto:[email protected]>> wrote:
>
> Jan & Ralf:
>
> I have a little good news, I have successfully gotten the non-root
> linux to display some startup output based on your suggestions.
> Also, there must have been some small difference between the stock
> linux-x86-demo cell config and the one I tweaked. I went back to
> the original and made some minor changes and it started displaying
> the serial output now for the non-root node.
>
> I ended up making the root cell the primary user of the ttyS0
> device, and just having the non-root linux output to the hypervisor
> virtual console (console=jailhouse). In my setup, there appears to
> be some process still attempting to use the ttyS0 randomly on the
> root linux, and that causes the hypervisor to halt it if I don't
> leave it in the PIO array. Not sure who is doing it.
>
>
> This is where i'm at now. The non-root linux keeps running into
> points where it tries to access PIO and gets parked:
>
> 1. First crash:
>
> [ 0.424925]kworker/u6:0 (27) used greated stack depth: 14656
> bytes left
> [ 0.425807]futex has table entries: 1024 (order: 4, 65536 bytes)
> "FATAL: Invalid PIO write, port: 70: size 1"
> ...
> Parking CPU 3 (Cell: "linux-x86-demo")
>
> According to /proc/ioports, 70 has to do with the Realtime clock
> (rtc0). I was able to temporarily add the 70 mapping to the
> non-root linux, but should it be there at all? I tried to disable
> RTC support in the guest kernel config.
>
>
> 2. On the next attempt, I then got further with a crash trying to
> initialize Ser device ttyS0:
>
> "[ 0.956146]Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled"
> "FATAL: Invalid PIO read, port: 3f9: size 1"
> ...
> Parking CPU 3 (Cell: "linux-x86-demo")
>
> I can get around that if I allow the non-root access in its pio
> table, but then I have a problem above with the root linux randomly
> trying to use it.
>
>
> 3. Now with the temporary adjustments to the non-root pio table
> above I get here:
>
> "[ 0.972399]clocksource:Switched to clocksource tsc"
> "FATAL: Invalid PIO read, port: 87: size 1"
> ...
> Parking CPU 3 (Cell: "linux-x86-demo")
>
> According to proc/ioports this one has to do with "dma page
> request". It isn't mapped in my root linux PIO or non-root linux array.
>
>
> Aside from the serial conflict, it seems like these should remain
> controlled by the root linux or hypervisor. Do you use a
> particular bare minimum x86 kernel config on your machine for the
> guest to get around these? I know Jan mentioned the
> jailhouse-images project might have something, but I couldn't find
> it digging around quick.
>
> Thanks to you both for your responses and patience. I got the root
> linux up and going on my own, but the non-root guest is proving to
> be more difficult.
>
> Wayne
>
>
>
> On Tue, Jun 11, 2019 at 11:31 AM Ralf Ramsauer
> <[email protected]
> <mailto:[email protected]>> wrote:
>
> Hi,
>
> On 6/10/19 7:45 PM, Wayne wrote:
> > Thank you for your responses Ralf and Jan.
> >
> > Yes, I have successfully gotten both the apic-demo and
> tiny-demo cells
> > to work on my system before attempting to load the non-root
> linux. I
> > can see serial output from them.
> >
> > However, if I try to share the UART between root and inmate
> for them I
> > run into a crash on the hypervisor if the root linux attempts
> to write
> > to the UART (after creating/starting a demo). Does the root
> linux lose
> > access once an inmate is created/started? Or am I missing
> something
> > from my System config or tiny-demo config that allows them to
> share? I
> > attached the crash in hypervisor_output.txt.
> >
> > Also, I noticed the tiny-demo output on the UART gets mirrored
> on the
>
> Yes, according to your config,
> JAILHOUSE_CELL_VIRTUAL_CONSOLE_ACTIVE is
> set. This means that the guest will use the virtual console. The
> virtual
> console uses the hypervisor output which depends whatever is set
> in your
> system configuration. In your case the hypervisor uses 3f8.
>
> And additionally, the inmate also a valid .console set. Again,
> 3f8. This
> means, it will use the console *and* the hypervisor debug
> hypercall. In
> your case, both outputs are routed to the same device which
> explains the
> mirroring.
>
> > Jailhouse virtual console. Does that mean that its writing to the
> > serial through the hypervisor and not directly accessing ttyS0
> itself?
>
> Exactly.
>
> > If so, is there a way to create the cell to write ttyS0
> directly as a
> > test? To see if it exhibits same behavior as the non-root linux.
>
> Sure, simply align the .address field of .console, and allow
> access to
> that port via .pio_bitmap.
>
> >
> > FYI, I create the tiny-demo like this:
> >
> > jailhouse cell create configs/x86/tiny-demo.cell
> > jailhouse cell load tiny-demo inmates/demos/x86/tiny-demo.cell
> > jailhouse cell start tiny-demo
> >
> > Going back to my original non-root linux question, I don't see any
> > output when connected over ttyS0. Its address is 0x3f8
> ("serial 1") in
> > the PIO, and that seems to work fine for the other simple
> demos, minus
> > the hypervisor crash issue above when sharing. I know there
> is the
> > other "serial 2" port but I haven't tried to use that one.
>
> I see that port on our Dell server as well. It's present, but I
> don't
> know where it's connected to.
>
> Let me just summarise your issue:
>
> You only have one serial line available, right? From the root-cell's
> POV, it's ttyS0 on 3f8.
>
> I would propose to share ttyS0 between the hypervisor and non-root
> Linux. This means:
>
> - Don't let the root-cell use ttyS0. Remove any console=ttyS0
> parameters from you commandline. Ensure that noone else accesses
> ttyS0 (e.g., getty@ttyS0 or other friends)
>
> - Set 3f8 as debug_console in your master.c (seems you already
> have)
>
> - Allow 3f8 access in the linux-demo.c (already set in the default
> linux-demo
>
> - Disallow 2f8 access in linux-demo.c:
> - [ 0x2f8/8 ... 0x2ff/8] = 0, /* serial2 */
> + [ 0x2f8/8 ... 0x2ff/8] = -1, /* serial2 */
>
> I don't know how Linux enumerates serial devices, but this
> ensures
> that Linux won't see the unconnected serial line and map
> ttyS0 to
> 3f8
>
> >
> > non root linux launched with:
> > /tools/jailhouse cell linux configs/x86/linux-x86.cell
> > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200"
>
> Then, this should actually work...
>
> And if not, then console=jailhouse0 should work.
>
> >
> > I can try the 4.19 siemens kernel and "next" branch for
> jailhouse you
> > suggested. Do you think that combination will resolve the missing
> > serial for the non-root linux? Otherwise, did my kernel
> config, system
> > config, and non-root linux cell configs look OK?
>
> I didn't look at it, but we should at least - even if some flags are
> still incorrect - see the "Uncompressing kernel..." thing.
>
> Wait -- one thing you could try: Deactivate CONFIG_EFI and
> especially
> CONFIG_EFI_STUBS. I'm not sure, but it could be that EFI_STUBS
> change
> the header format that our bootloader patches.
>
> >
> > Thanks again for your help.
>
> No problem. But it's not yet working. ;-)
>
> Ralf
>
> >
> >
> > On Fri, Jun 7, 2019 at 5:30 PM Ralf Ramsauer
> > <[email protected]
> <mailto:[email protected]>
> > <mailto:[email protected]
> <mailto:[email protected]>>> wrote:
> >
> > Hi Wayne,
> >
> > On 6/7/19 10:04 PM, Wayne wrote:
> > > Hi Ralf,
> > >
> > > Thank you for your responses. I have attached the
> requested config
> > > files. I am now just trying to use the same bzImage for
> the root and
> > > the non-root linux node.
> >
> > One question that you didn't answer: do apic-demo or
> tiny-demo work?
> > They need to to have correct .console parameter set, so
> ensure that
> > parameters are set if you run those demos in your
> linux-demo cell. Just
> > compare it to tiny-demo.c or apic-demo.c
> >
> > >
> > > Yes, I am using vanilla 4.16 because the documentation
> indicated
> > that it
> > > is the first mainline release to officially support
> Jailhouse Guest
> > > images. Do you suggest that I use something else?
> >
> > As Jan already wrote: Try to use Siemens' 4.19 branch.
> >
> > >
> > > Right now i'm just trying to get some serial output from
> the non-root
> > > Linux. Hopefully, I have that configured correctly.
> Ideally, I would
> > > like to share the same serial console for the Root node
> and non-root
> > > node eventually if possible. I have been OK with using
> the virtual
> > > console (jailhouse console -f) for the Hypervisor output.
> >
> > In addition to Jan's comment:
> >
> > Currently you only have access to the jailhouse console
> via 'jailhouse
> > console' from your root cell. Imagine your root cell
> crashes for some
> > reason. You will never see the fault reason, which makes
> things hard to
> > debug.
> >
> > I would suggest to configure the hypervisor to use the
> serial console.
> > You can share the device with the non-root Linux, that's
> no problem.
> >
> > In this case, the non-root kernel's output will always be
> printed to the
> > serial console. Directly, if you choose console=ttySx, and
> indirectly
> > via the hypervisor if you choose console=jailhouse.
> >
> > BTW: According to your config, your system is a PowerEdge,
> and the
> > non-root cell gets both, 0x2f8 and 0x3f8. Are you sure
> that ttyS1 is the
> > correct console that you are connected to?
> >
> > Just mentioning this as we have a similar machine here,
> and, afair, both
> > platform serial device are 'usable', but one ends in the
> nirvana while
> > it is accessible.
> >
> > Ah, and one last thing: try to switch to the current next
> branch for
> > jailhouse. Jan just integrated a few patches from me that
> might also be
> > relevant for your machine.
> >
> > HTH
> > Ralf
> >
> > >
> > > Wayne
> > >
> > > On Fri, Jun 7, 2019 at 9:10 AM Ralf Ramsauer
> > > <[email protected]
> <mailto:[email protected]>
> > <mailto:[email protected]
> <mailto:[email protected]>>
> > > <mailto:[email protected]
> <mailto:[email protected]>
> > <mailto:[email protected]
> <mailto:[email protected]>>>> wrote:
> > >
> > > Hi,
> > >
> > > On 6/7/19 2:28 PM, Wayne wrote:
> > > > Hello,
> > > >
> > > > I am new to Linux development and Jailhouse. I have
> > successfully
> > > booted
> > > > the Jailhouse Hypervisor and root cell on a bare
> metal X86 Linux
> > > system
> > > > (No QEMU). I am now trying to load a non-root
> Linux cell
> > and I have a
> > > > few questions. Jailhouse accepts and starts my
> non-root
> > linux cell
> > > > configuration and I see it as "running" through the
> > "jailhouse cell
> > > > list" command. However, I don't see any serial
> output from the
> > > > "non-root linux" cell booting up. I’m not sure
> what the
> > non-root node
> > > > is doing at this point.
> > >
> > > Ok, first of all, can you see any output from
> Jailhouse's demo
> > inmates
> > > (e.g., tiny-demo or apic-demo)?
> > >
> > > >
> > > > My root node is a 4.16 kernel
> configured this
> > way:
> > > > 1.
> CONFIG_JAILHOUSE_GUEST is
> > not set
> > > > 2. CONFIG_SERIO=y
> > > > 3.
> > CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> > > >
> > > > My non-root node is a 4.16 kernel
> configured
> > this way:
> > > > 1.
> CONFIG_JAILHOUSE_GUEST=y
> > > > 2. CONFIG_SERIO=m
> (can't seem
> > to disable
> > > > completely in my config for 4.16)
> > > > 3.
> > CONFIG_SERIAL_8250_RUNTIME_UARTS=1
> > >
> > > Could you please provide a full .config? What branch
> are you
> > exactly
> > > using? Vanilla 4.16?
> > >
> > > Please attach your system config and your non-root linux
> > config as well.
> > >
> > > You may also want to try
> https://github.com/siemens/linux . See
> > > jailhouse-enabling branches for some releases.
> > >
> > > >
> > > > In general, do the kernel config settings have to
> match between
> > > the root
> > > > node and non-linux or is the above fine?
> > >
> > > No, they do not have to be the same, but they can.
> Still, an
> > analysis
> > > requires your .config.
> > >
> > > >
> > > > The vmlinux-4.1.16-Guest bzImage is approx 7MB,
> and the
> > inmate node is
> > > > allocated ~75MB of RAM.
> > > >
> > > > I have a single UART, so I have configured the
> root cell
> > system config
> > > > to output to the virtual hypervisor console:
> > >
> > > Wait... You configured your root-cell to use the
> hypervisor debug
> > > console? How? It's only available once the hypervisor is
> > enabled. How
> > > should this work?
> > >
> > > But let's have a look at your configuration first.
> > >
> > > So you want to:
> > > - Have the UART (0x3f8) available in the non-root cell
> > > - Use 0x3f8 as hypervisor debug console as well
> > > - No console for root-cell
> > >
> > > Did I get this right?
> > >
> > > Ralf
> > >
> > > >
> > > > .flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE,
> > > >
> > > > .debug_console = {
> > > > .type = JAILHOUSE_CON_TYPE_NONE,
> > > > },
> > > >
> > > > and I have configured the non-root linux cell to
> output to
> > the UART:
> > > >
> > > > (Added serial 0x3f8 to pio bitmap for non-root
> linux) and
> > started the
> > > > node with this:
> > > >
> > > > ./tools/jailhouse cell linux
> configs/x86/linux-x86.cell
> > > > /boot/vmlinux-4.1.16-Guest -c "console=ttyS0,115200"
> > > > (Note I also tried "console=jailhouse" in the
> command above,
> > and that
> > > > produces the same result)
> > > >
> > > > I then see the following on my hypervisor console
> > (./tools/jailhouse
> > > > console -f):
> > > >
> > > > Created cell "linux-x86-demo"
> > > > ...
> > > > Cell "linux-x86-demo" can be loaded
> > > > Started cell "linux-x86-demo"
> > > >
> > > > After a little while I do get a parked CPU error
> on the root
> > node,
> > > looks
> > > > like its trying to do something with the UART as well:
> > > > FATAL: Invalid PIO read, port: 3fe size: 1
> > > >
> > > > I would expect something to pop out on the UART
> from the
> > non-root
> > > linux
> > > > node first. Note that root node has serial 0x3f8
> disabled
> > in its pio
> > > > bitmap.
> > > >
> > > > I verifed that the UART is functioning by allowing the
> > hypervisor to
> > > > print to it and also performed an echo test over
> ttyS0.
> > > >
> > > > I have tried several configurations of
> kernel.....including your
> > > current
> > > > "queues/jailhouse" branch head kernel for the
> non-root node,
> > along
> > > with
> > > > the kernel config for 4.7 posted in this thread
> below (but I
> > get same
> > > > result as above when I start it, no kernel output):
> > > >
> > > >
> > >
> >
>
> "https://groups.google.com/forum/#!searchin/jailhouse-dev/Re$3A$20Failed$20to$20boot$20jailhouse%7Csort:relevance/jailhouse-dev/M7UO89XFIk0/Qi40DDuMBAAJ";.
> > > >
> > > > Any information you can provide to me will be
> helpful. I'm
> > not sure
> > > > what might be going wrong here.
> > > >
> > > > Thanks,
> > > > Wayne
> > > >
> > > > --
> > > > You received this message because you are
> subscribed to the
> > Google
> > > > Groups "Jailhouse" group.
> > > > To unsubscribe from this group and stop receiving
> emails
> > from it, send
> > > > an email to
> [email protected]
> <mailto:jailhouse-dev%[email protected]>
> > <mailto:jailhouse-dev%[email protected]
> <mailto:jailhouse-dev%[email protected]>>
> > > <mailto:jailhouse-dev%[email protected]
> <mailto:jailhouse-dev%[email protected]>
> > <mailto:jailhouse-dev%[email protected]
> <mailto:jailhouse-dev%[email protected]>>>
> > > > <mailto:[email protected]
> <mailto:jailhouse-dev%[email protected]>
> > <mailto:jailhouse-dev%[email protected]
> <mailto:jailhouse-dev%[email protected]>>
> > > <mailto:jailhouse-dev%[email protected]
> <mailto:jailhouse-dev%[email protected]>
> > <mailto:jailhouse-dev%[email protected]
> <mailto:jailhouse-dev%[email protected]>>>>.
> > > > To view this discussion on the web visit
> > > >
> > >
> >
>
> https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com
> > > >
> > >
> >
>
> <https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc2iKk1J6%2B0huh5__dS4HyujXzV9r%2BLbKLzuVZ4K3Bt5eA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> > > > For more options, visit
> https://groups.google.com/d/optout.
> > >
> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "Jailhouse" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc0O2zSeuLLY3MaeRW7cQrWbq-6Y2BHJg%2Bx_j6nk%3DECa_A%40mail.gmail.com
> <https://groups.google.com/d/msgid/jailhouse-dev/CA%2B%2BKhc0O2zSeuLLY3MaeRW7cQrWbq-6Y2BHJg%2Bx_j6nk%3DECa_A%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/ad13aaa6-41ac-14fd-33fe-fa4f6dfc789e%40oth-regensburg.de.
For more options, visit https://groups.google.com/d/optout.
