Kudos Waldek!
On Thu, May 21, 2020 at 1:58 PM Waldek Kozaczuk <[email protected]>
wrote:
>
> *ubuntu@ubuntu-rasp4*:*~*$ uname -a
>
> Linux ubuntu-rasp4 5.3.0-1025-raspi2 #27-Ubuntu SMP Fri May 8 08:32:04
> UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
>
> *ubuntu@ubuntu-rasp4*:*~*$ ./firecracker-v0.21.1-aarch64 --no-api --config
> -file ./osv-config.json
>
> OSv v0.55.0-9-g840428ad
>
> PSCI: version 65536.0 detected.
>
> setup_arm_clock() ENTERED, lr=00000000800d54e0
>
> arm_clock(): frequency read as 000000000337f980
>
> interrupt_table::interrupt_table() ENTERED, lr=00000000800a8d1c
>
> gic_driver::init_cpu() ENTERED, lr=0000000080201c80
>
> CPU interface enabled.
>
> gic_driver::init_dist() ENTERED, lr=0000000080201c8c
>
> number of supported IRQs: 0000000000000080
>
> interrupt table: gic driver created.
>
> registered IRQ id=0000000000000004
>
> registered IRQ id=000000000000001b
>
> registered IRQ id=0000000000000000
>
> Premain complete!
>
> smp_launch ENTERED, lr=00000000800d709c
>
> Booted up in 0.00 ms
>
> Cmdline: console=ttyS0 --verbose --nomount --maxnic=0 /tools/hello.so
> earlycon=uart,mmio,0x40001000
>
> faulting address 000010000001fea0
>
> faulting address 0000100000000328
>
> faulting address 0000100000020000
>
> faulting address 0000100000050018
>
> faulting address 00001000000301c8
>
> faulting address 000010000004fe00
>
> Hello from C code
>
> 2020-05-21T20:24:04.080765641 [anonymous-instance:ERROR:src/vmm/src/vstate
> .rs:951] Unexpected exit reason on vcpu run: SystemEvent
>
> In general it took me a bit of research as I am not really familiar with
> ARM architecture and even reading the assembly was a bit a challenge to say
> the least. And then debugging without debugger and any console (:-( .. so
> not debug() for long time. But all in all it was not too bad and the
> changes that I had to make to OSv are in my opinion much smaller and easier
> comparing to x86_64.
>
> Below you will see the "hack-patch" showing what changes I had to make.
> Logically, following things had to be changed:
>
> - The most important thing was to move kernel from 0x40000000 (1GB) to
> 0x80000000 (2GB) which required changing one line in Makefile (see below)
> and changing boot paging table to map the 2GB-3GB area of memory; only then
> I could actually start debugging :-) I wonder if it will also work on
> QEMU/KVM - possinly qemu boot loader will inspect ELF and place it
> accordingly in memory; firecracker does not read ELF and simply places it
> at 2GB
> - To get console working I guessed that that need to create equivalent
> class for isa_serial_console but communicating over mmio (see
> mmio_isa_serial_console.hh/cc) which in essence invokes mmio_set*/mmio_get*
> (it would be nice to extract common code somehow - suggestions welcome)
> - Some more trivial changes - for now mostly disabling things - for
> now quite disorganized
>
>
> diff --git a/Makefile b/Makefile
> index db3c68cf..ffd570a5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -354,6 +354,7 @@ tools := tools/mkfs/mkfs.so tools/cpiod/cpiod.so
> $(out)/tools/%.o: COMMON += -fPIC
>
> tools += tools/uush/uush.so
> +tools += tools/uush/hello.so
> tools += tools/uush/ls.so
> tools += tools/uush/mkdir.so
>
> @@ -451,8 +452,8 @@ endif # x64
>
> ifeq ($(arch),aarch64)
>
> -kernel_base := 0x40080000
> -kernel_vm_base := 0x40080000
> +kernel_base := 0x80080000
> +kernel_vm_base := 0x80080000
> app_local_exec_tls_size := 0x0
>
> include $(libfdt_base)/Makefile.libfdt
> @@ -816,6 +817,7 @@ drivers += drivers/xenplatform-pci.o
> endif # x64
>
> ifeq ($(arch),aarch64)
> +drivers += drivers/mmio-isa-serial.o
> drivers += drivers/pl011.o
> drivers += drivers/xenconsole.o
> drivers += drivers/virtio.o
> diff --git a/arch/aarch64/arch-dtb.cc b/arch/aarch64/arch-dtb.cc
> index b59f1dcc..cd0719e8 100644
> --- a/arch/aarch64/arch-dtb.cc
> +++ b/arch/aarch64/arch-dtb.cc
> @@ -225,7 +225,8 @@ bool dtb_get_gic_v2(u64 *dist, size_t *dist_len, u64
> *cpu, size_t *cpu_len)
> if (!dtb)
> return false;
>
> - node = fdt_node_offset_by_compatible(dtb, -1, "arm,cortex-a15-gic");
> + //node = fdt_node_offset_by_compatible(dtb, -1, "arm,cortex-a15-gic");
> + node = fdt_node_offset_by_compatible(dtb, -1, "arm,gic-400");
> if (node < 0)
> return false;
>
> @@ -613,11 +614,13 @@ void __attribute__((constructor(init_prio::dtb)))
> dtb_setup()
> abort("dtb_setup: failed to parse cpu mpid.\n");
> }
>
> + //TODO: Parse PCI only if required and
> + // abort if missing only if required
> dtb_timer_irq = dtb_parse_timer_irq();
> dtb_pci_irqmask = dtb_parse_pci_irqmask();
> dtb_pci_irqmap_count = dtb_parse_pci_irqmap_count();
> if (!dtb_parse_pci_irqmap(dtb_pci_bdfs, dtb_pci_irq_ids,
> dtb_pci_irqmap_count)) {
> - abort("dtb_setup: failed to parse pci_irq_map.\n");
> + //abort("dtb_setup: failed to parse pci_irq_map.\n");
> }
>
> register u64 edata;
> diff --git a/arch/aarch64/arch-setup.cc b/arch/aarch64/arch-setup.cc
> index bdbb1266..b12546ae 100644
> --- a/arch/aarch64/arch-setup.cc
> +++ b/arch/aarch64/arch-setup.cc
> @@ -17,6 +17,7 @@
> #include <osv/debug.hh>
> #include <osv/commands.hh>
> #include <osv/xen.hh>
> +#include <osv/version.h>
>
> #include "arch-mmu.hh"
> #include "arch-dtb.hh"
> @@ -25,6 +26,7 @@
> #include "drivers/pl011.hh"
> #include "early-console.hh"
> #include <osv/pci.hh>
> +#include "drivers/mmio-isa-serial.hh"
>
> #include <alloca.h>
>
> @@ -93,9 +95,10 @@ void arch_setup_free_memory()
>
> if (!is_xen()) {
> /* linear_map [TTBR0 - UART] */
> - addr = (mmu::phys)console::aarch64_console.pl011.get_base_addr();
> - mmu::linear_map((void *)addr, addr, 0x1000, mmu::page_size,
> - mmu::mattr::dev);
> + //TODO: Only run it if pl011 is detected
> + //addr =
> (mmu::phys)console::aarch64_console.pl011.get_base_addr();
> + //mmu::linear_map((void *)addr, addr, 0x1000, mmu::page_size,
> + // mmu::mattr::dev);
> }
>
> /* linear_map [TTBR0 - GIC DIST and GIC CPU] */
> @@ -110,12 +113,17 @@ void arch_setup_free_memory()
> mmu::linear_map((void *)cpu, (mmu::phys)cpu, cpu_len, mmu::page_size,
> mmu::mattr::dev);
>
> - arch_setup_pci();
> + //TODO: Call it only if PCI available
> + //arch_setup_pci();
>
> // get rid of the command line, before memory is unmapped
> osv::parse_cmdline(cmdline);
>
> mmu::switch_to_runtime_page_tables();
> +
> + //TODO: This is probably wrong place to init the console
> + arch_init_early_console();
> + debug_early("OSv " OSV_VERSION "\n");
> }
>
> void arch_setup_tls(void *tls, const elf::tls_data& info)
> @@ -185,27 +193,35 @@ void arch_init_early_console()
> return;
> }
>
> + /* Comment out for now
> new (&console::aarch64_console.pl011) console::PL011_Console();
> console::arch_early_console = console::aarch64_console.pl011;
> int irqid;
> - u64 addr = dtb_get_uart(&irqid);
> - if (!addr) {
> + u64 addr = dtb_get_uart(&irqid);*/
> + //if (!addr) {
> /* keep using default addresses */
> - return;
> - }
> + // return;
> + //}
>
> - console::aarch64_console.pl011.set_base_addr(addr);
> - console::aarch64_console.pl011.set_irqid(irqid);
> + //console::aarch64_console.pl011.set_base_addr(addr);
> + //console::aarch64_console.pl011.set_irqid(irqid);
> +
> + new (&console::aarch64_console.mmio_isa_serial)
> console::mmio_isa_serial_console();
> + console::arch_early_console =
> console::aarch64_console.mmio_isa_serial;
> +
> + console::mmio_isa_serial_console::early_init();
> }
>
> bool arch_setup_console(std::string opt_console)
> {
> + /*
> if (opt_console.compare("pl011") == 0) {
> console::console_driver_add(&console::arch_early_console);
> } else if (opt_console.compare("all") == 0) {
> console::console_driver_add(&console::arch_early_console);
> } else {
> return false;
> - }
> + }*/
> + console::console_driver_add(&console::arch_early_console);
> return true;
> }
> diff --git a/arch/aarch64/boot.S b/arch/aarch64/boot.S
> index b29fd16a..1d2af75c 100644
> --- a/arch/aarch64/boot.S
> +++ b/arch/aarch64/boot.S
> @@ -188,21 +188,15 @@ ident_pt_l4_ttbr0:
> .quad 0
> .endr
> ident_pt_l3_ttbr0:
> - .quad ident_pt_l2_0_ttbr0 + 0x3
> - .quad ident_pt_l2_1_ttbr0 + 0x3
> - .rept 510
> + .quad 0
> + .quad 0
> + .quad ident_pt_l2_2_ttbr0 + 0x3 // Map 4GB-6GB one-to-one
> + .rept 509
> .quad 0
> .endr
> -ident_pt_l2_0_ttbr0:
> - index = 0
> - offset = 0x00000000
> - .rept 512
> - .quad (index << 21) + 0x401
> - index = index + 1
> - .endr
> -ident_pt_l2_1_ttbr0:
> +ident_pt_l2_2_ttbr0:
> index = 0
> - offset = 0x40000000
> + offset = 0x80000000
> .rept 512
> .quad offset + (index << 21) + 0x411
> index = index + 1
> diff --git a/arch/aarch64/early-console.cc b/arch/aarch64/early-console.cc
> index d5196c74..8c91c965 100644
> --- a/arch/aarch64/early-console.cc
> +++ b/arch/aarch64/early-console.cc
> @@ -13,6 +13,7 @@
> namespace console {
>
> union AARCH64_Console aarch64_console;
> -console_driver & arch_early_console = aarch64_console.pl011;
> +//console_driver & arch_early_console = aarch64_console.pl011;
> +console_driver & arch_early_console = aarch64_console.mmio_isa_serial;
>
> }
> diff --git a/arch/aarch64/early-console.hh b/arch/aarch64/early-console.hh
> index cbc2cac4..cd39738b 100644
> --- a/arch/aarch64/early-console.hh
> +++ b/arch/aarch64/early-console.hh
> @@ -11,12 +11,15 @@
> #include <drivers/console-driver.hh>
> #include <drivers/pl011.hh>
> #include <drivers/xenconsole.hh>
> +#include <drivers/mmio-isa-serial.hh>
> +
>
> namespace console {
>
> union AARCH64_Console {
> PL011_Console pl011;
> XEN_Console xen;
> + mmio_isa_serial_console mmio_isa_serial;
>
> AARCH64_Console() {}; /* placement new is used to initialize object
> */
> ~AARCH64_Console() {}; /* won't ever be called */
> diff --git a/arch/aarch64/mmu.cc b/arch/aarch64/mmu.cc
> index fef48e5a..012d3c57 100644
> --- a/arch/aarch64/mmu.cc
> +++ b/arch/aarch64/mmu.cc
> @@ -18,11 +18,11 @@ void page_fault(exception_frame *ef)
> {
> sched::fpu_lock fpu;
> SCOPE_LOCK(fpu);
> - debug_early_entry("page_fault");
> + //debug_early_entry("page_fault");
> u64 addr;
> asm volatile ("mrs %0, far_el1" : "=r"(addr));
> debug_early_u64("faulting address ", (u64)addr);
> - debug_early_u64("elr exception ra ", (u64)ef->elr);
> + //debug_early_u64("elr exception ra ", (u64)ef->elr);
>
> if (fixup_fault(ef)) {
> debug_early("fixed up with fixup_fault\n");
> @@ -45,7 +45,7 @@ void page_fault(exception_frame *ef)
> mmu::vm_fault(addr, ef);
> }
>
> - debug_early("leaving page_fault()\n");
> + //debug_early("leaving page_fault()\n");
> }
>
> namespace mmu {
> diff --git a/drivers/mmio-isa-serial.cc b/drivers/mmio-isa-serial.cc
> new file mode 100644
> index 00000000..c59cec8d
> --- /dev/null
> +++ b/drivers/mmio-isa-serial.cc
> @@ -0,0 +1,134 @@
> +/*
> + * Copyright (C) 2013 Cloudius Systems, Ltd.
> + *
> + * This work is open source software, licensed under the terms of the
> + * BSD license as described in the LICENSE file in the top-level
> directory.
> + */
> +
> +#include "mmio-isa-serial.hh"
> +
> +namespace console {
> +
> +// UART registers, offsets to ioport:
> +enum regs {
> + IER = 1, // Interrupt Enable Register
> + FCR = 2, // FIFO Control Register
> + LCR = 3, // Line Control Register
> + MCR = 4, // Modem Control Register
> + LSR = 5, // Line Control Register
> + MSR = 6, // Modem Status Register
> + SCR = 7, // Scratch Register
> + DLL = 0, // Divisor Latch LSB Register
> + DLM = 1, // Divisor Latch MSB Register
> +};
> +
> +enum lcr {
> + // When bit 7 (DLAB) of LCR is set to 1, the two registers 0 and 1
> + // change their meaning and become two bytes controlling the baud rate
> + DLAB = 0x80, // Divisor Latch Access Bit in LCR register
> + LEN_8BIT = 3,
> +};
> +
> +// Various bits of the Line Status Register
> +enum lsr {
> + RECEIVE_DATA_READY = 0x1,
> + OVERRUN = 0x2,
> + PARITY_ERROR = 0x4,
> + FRAME_ERROR = 0x8,
> + BREAK_INTERRUPT = 0x10,
> + TRANSMIT_HOLD_EMPTY = 0x20,
> + TRANSMIT_EMPTY = 0x40,
> + FIFO_ERROR = 0x80,
> +};
> +
> +// Various bits of the Modem Control Register
> +enum mcr {
> + DTR = 0x1,
> + RTS = 0x2,
> + AUX_OUTPUT_1 = 0x4,
> + AUX_OUTPUT_2 = 0x8,
> + LOOPBACK_MODE = 0x16,
> +};
> +
> +mmioaddr_t mmio_isa_serial_console::_addr_mmio;
> +
> +void mmio_isa_serial_console::early_init()
> +{
> + u64 address = 0x40001000; //TODO: Should parse from boot command line
> ('earlycon=uart,mmio,0x40001000')
> + u64 size = 4096;
> +
> + _addr_mmio = mmio_map(address, size);
> +
> + // Set the UART speed to to 115,200 bps, This is done by writing 1,0
> to
> + // Divisor Latch registers, but to access these we need to temporarily
> + // set the Divisor Latch Access Bit (DLAB) on the LSR register,
> because
> + // the UART has fewer ports than registers...
> + mmio_setb(_addr_mmio + (int)regs::LCR, lcr::LEN_8BIT | lcr::DLAB);
> + mmio_setb(_addr_mmio + (int)regs::DLL, 1);
> + mmio_setb(_addr_mmio + (int)regs::DLM, 0);
> + mmio_setb(_addr_mmio + (int)regs::LCR, lcr::LEN_8BIT);
> +
> + // interrupt threshold
> + mmio_setb(_addr_mmio + (int)regs::FCR, 0);
> +
> + // disable interrupts
> + mmio_setb(_addr_mmio + (int)regs::IER, 0);
> +
> + // Most physical UARTs need the MCR AUX_OUTPUT_2 bit set to 1 for
> + // interrupts to be generated. QEMU doesn't bother checking this
> + // bit, but interestingly VMWare does, so we must set it.
> + mmio_setb(_addr_mmio + (int)regs::MCR, mcr::AUX_OUTPUT_2);
> +}
> +
> +void mmio_isa_serial_console::write(const char *str, size_t len)
> +{
> + while (len-- > 0)
> + putchar(*str++);
> +}
> +
> +bool mmio_isa_serial_console::input_ready()
> +{
> + u8 val = mmio_getb(_addr_mmio + (int)regs::LSR);
> + // On VMWare hosts without a serial port, this register always
> + // returns 0xff. Just ignore it instead of spinning incessantly.
> + return (val != 0xff && (val & lsr::RECEIVE_DATA_READY));
> +}
> +
> +char mmio_isa_serial_console::readch()
> +{
> + u8 val;
> + char letter;
> +
> + do {
> + val = mmio_getb(_addr_mmio + (int)regs::LSR);
> + } while (!(val & (lsr::RECEIVE_DATA_READY | lsr::OVERRUN |
> lsr::PARITY_ERROR | lsr::FRAME_ERROR)));
> +
> + letter = mmio_getb(_addr_mmio);
> +
> + return letter;
> +}
> +
> +void mmio_isa_serial_console::putchar(const char ch)
> +{
> + u8 val;
> +
> + do {
> + val = mmio_getb(_addr_mmio + (int)regs::LSR);
> + } while (!(val & lsr::TRANSMIT_HOLD_EMPTY));
> +
> + mmio_setb(_addr_mmio, ch);
> +}
> +
> +void mmio_isa_serial_console::enable_interrupt()
> +{
> + // enable interrupts
> + mmio_setb(_addr_mmio + (int)regs::IER, 1);
> +}
> +
> +void mmio_isa_serial_console::dev_start() {
> + //TODO: Figure out which interrupt and what kind to use
> + //_irq.reset(new sgi_edge_interrupt(4, [&] { _thread->wake(); }));
> + enable_interrupt();
> +}
> +
> +}
> diff --git a/drivers/mmio-isa-serial.hh b/drivers/mmio-isa-serial.hh
> new file mode 100644
> index 00000000..f422dfda
> --- /dev/null
> +++ b/drivers/mmio-isa-serial.hh
> @@ -0,0 +1,39 @@
> +/*
> + * Copyright (C) 2013 Cloudius Systems, Ltd.
> + *
> + * This work is open source software, licensed under the terms of the
> + * BSD license as described in the LICENSE file in the top-level
> directory.
> + */
> +
> +#ifndef DRIVERS_MMIO_ISA_SERIAL_HH
> +#define DRIVERS_MMIO_ISA_SERIAL_HH
> +
> +#include "console-driver.hh"
> +#include <osv/pci.hh>
> +#include <osv/sched.hh>
> +#include <osv/interrupt.hh>
> +#include <osv/mmio.hh>
> +
> +namespace console {
> +
> +class mmio_isa_serial_console : public console_driver {
> +public:
> + static void early_init();
> + virtual void write(const char *str, size_t len);
> + virtual void flush() {}
> + virtual bool input_ready() override;
> + virtual char readch();
> +
> +private:
> + //std::unique_ptr<sgi_edge_interrupt> _irq;
> + static mmioaddr_t _addr_mmio;
> +
> + virtual void dev_start();
> + void enable_interrupt();
> + static void putchar(const char ch);
> + virtual const char *thread_name() { return "mmio-isa-serial-input"; }
> +};
> +
> +}
> +
> +#endif
> diff --git a/loader.cc b/loader.cc
> index 66bfb52c..9867ca49 100644
> --- a/loader.cc
> +++ b/loader.cc
> @@ -94,12 +94,12 @@ extern "C" {
>
> void premain()
> {
> - arch_init_early_console();
> + //arch_init_early_console();
>
> /* besides reporting the OSV version, this string has the function
> to check if the early console really works early enough,
> without depending on prior initialization. */
> - debug_early("OSv " OSV_VERSION "\n");
> + //debug_early("OSv " OSV_VERSION "\n");
>
> arch_init_premain();
>
> @@ -116,6 +116,7 @@ void premain()
> for (auto init = inittab.start; init < inittab.start + inittab.count;
> ++init) {
> (*init)();
> }
> + debug_early("Premain complete!\n");
> boot_time.event(".init functions");
> }
>
> @@ -230,6 +231,7 @@ static void parse_options(int loader_argc, char**
> loader_argv)
> opt_maxnic = true;
> maxnic = options::extract_option_int_value(options_values,
> "maxnic", handle_parse_error);
> }
> + maxnic = 0;
>
> if (extract_option_flag(options_values, "trace-backtrace")) {
> opt_log_backtrace = true;
> @@ -261,6 +263,7 @@ static void parse_options(int loader_argc, char**
> loader_argv)
> }
>
> opt_mount = !extract_option_flag(options_values, "nomount");
> + opt_mount = false;
> opt_pivot = !extract_option_flag(options_values, "nopivot");
> opt_random = !extract_option_flag(options_values, "norandom");
> opt_init = !extract_option_flag(options_values, "noinit");
> @@ -351,7 +354,8 @@ std::vector<std::vector<std::string> >
> prepare_commands(char* app_cmdline)
> bool ok;
>
> printf("Cmdline: %s\n", app_cmdline);
> - commands = osv::parse_command_line(app_cmdline, ok);
> + //commands = osv::parse_command_line(app_cmdline, ok);
> + commands = osv::parse_command_line("/tools/hello.so", ok);
>
> if (!ok) {
> puts("Failed to parse command line.");
> diff --git a/usr_ramfs.manifest.skel b/usr_ramfs.manifest.skel
> index 4bdfdd32..1a04c22f 100644
> --- a/usr_ramfs.manifest.skel
> +++ b/usr_ramfs.manifest.skel
> @@ -3,6 +3,8 @@
> /libvdso.so: libvdso.so
> /tools/mount-fs.so: tools/mount/mount-fs.so
> /tools/umount.so: tools/mount/umount.so
> +/tools/uush.so: tools/uush/uush.so
> +/tools/hello.so: tools/uush/hello.so
> /usr/lib/libgcc_s.so.1: %(libgcc_s_dir)s/libgcc_s.so.1
> /&/etc/hosts: ../../static/&
>
>
>
> Please not that I have not tested how it works with multiple CPUs or if
> virtual block and network devices operate.
>
> Waldek
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/f4c6cafe-690a-4619-b7b6-e414e3f9343a%40googlegroups.com
> <https://groups.google.com/d/msgid/osv-dev/f4c6cafe-690a-4619-b7b6-e414e3f9343a%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/CAKUaUn6s4bCRBiEMyx8ALpmarUj01Bh%3DHUwtusu6MXHnLgmNrQ%40mail.gmail.com.
