As you know OSv supports running unmodified Linux executable as long as they use subset of Linux API implemented by OSv. On other hand OSv kernel is not compatible with Linux as far as boot process goes. By that I mean one cannot boot OSv on hypervisors that support booting some form of Linux kernel - qemu with --kernel option, hyperkit with -kexec option and firecracker. The Linux boot protocol is quite involved and carries a lot of legacy so I advocate we only implement the minimal bits of it we need to run OSv on 3 mentioned hypervisors.
I also think it would be desirable to adhere to the following constraints: - keep minimum number of boot artifacts (usr.img, boot.bin, lzloader.elf, loader.elf, etc) - keep OSv boot process and artifacts backwards compatible with 16-bit mode on Xen, QEMU, VirtualBox, VMware (this really means that usr.img needs to stay bootable on those platforms but artifacts like lzloader.elf or loader.elf do not need to stay the same) Let me recap how OSv process works now (for details please look at these somewhat outdated Wiki pages - https://github.com/cloudius-systems/osv/wiki/OSv-early-boot-(MBR) and https://github.com/cloudius-systems/osv/wiki/OSv-lzloader-and-early-loader). The OSv boot process relies on following artifacts produced by build process: - loader.elf - 64-bit kernel ELF file with 32-bit entry point start32 - get placed at 0x200000 in memory by uncompress_kernel - start32 does following: - sets up GDT for 32-bit and 64-bit - sets up and enables paging - eventually switches to long mode (64-bit) by jumping to start64 - lzloader.elf - 32-bit ELF file that probably does not need to be an ELF (*somewhat similar to Linux'es vmlinuz file*?) - it is loaded by logic in boot.bin (boot16.S) from disk and placed at 0x100000 in memory - it is comprised of compressed loader.elf and fastlz code to decompress it chunk-by-chunk in "almost-in-place" fashion so it ends up at the address 0x200000 in memory - boot.bin - 512-bytes long 16-/32-bit binary produced from boot16.S that: - loads lzloader.elf from disk in 32K chunks at 0x100000 in memory - loads command line from disk into memory (part of multiboot info struct) - detects available memory and places info in multiboot info struct - switches to protected mode and calls uncompress_kernel (code in lzloader.elf) - finally calls start32 (code in loader.elf) - usr.img that is comprised of boot.bin, cmdline and lzloader.elf and looks like this: - first 512 bytes are boot.bin - - it is MBR (Master Boot Record) - from 513 until 64K-1 there is a command line followed with 0-s - at 64K offset (128 blocks of 512 bytes) starts lzloader.elf - - 0......512.................65536=64k..... - [boot.bin][cmdline]00000000[lzloader.elf] The Linux boot protocol is documented here - https://www.kernel.org/doc/Documentation/x86/boot.txt and I think we only care about "32-bit BOOT PROTOCOL" of it (required by qemu --kernel and hyperkit -kexec) and some subset of "64-bit BOOT PROTOCOL" (required by firecracker).\ In order to make OSv comply with either of these two Linux boot flavors, two aspects of it need to be considered: 1. The kernel loader file layout and what information goes in there to satisfy particular hypervisor when loading the kernel file. - in case of "32-bit BOOT PROTOCOL" the file is expected to be in "vmlinuz" format and have so called "zero page" information (struct boot_params) located at 0x1f1 offset (see https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/bootparam.h) - in case of firecracker the loader file needs to be a 64-bit ELF with 64-bit entry point - almost like OSv loader.elf except loader.elf has 32-bit entry point 2. The information that is passed by hypervisor to the Linux guest (command line and memory info) and the VM state hypervisor sets before calling kernel entry point. - the command line and memory info gets set in proper fields of boot_params struct field; the boot_params location in memory is specified in esi/rdi register - in case of "32-bit BOOT PROTOCOL" the VM is set to 32-bit mode and in general its states resembles roughly what start32 in OSv expects I think - in case of firecracker the VM is set to 64-bit mode with GDT and page tables setup Linux-way (please see this diff - https://github.com/wkozaczuk/osv/commit/3628f5607ae9dee8d812456571360f32e3c1aa29#diff-ee660a0822ff34de6d00ee9b0c382ce8- to see changes I had to make - just a hack not a final solution - to make OSv loader.elf to boot on firecracker) So given all that I was wondering what are you thoughts/ideas of making OSv be Linux-boot compliant enough so it boots on firecracker, QEMU --kernel and hyperkit -exec? Here are my rough ideas: 1. Convert lzloader.elf to lzloader.bin that has enough of vmlinuz format (zero page). 2. Optionally introduce not-compressed version of lzloader.bin. 3. Somehow enhance loader.elf to support both 32-bit and 64-bit boot. This seems impossible as ELF can only have one entry point. - Maybe move protected mode setup logic from boot.S to lzloader.bin and make start64 the entry point of loader.elf? - Or maybe make lzloader.bin jump to start32 but make start32 no longer an entry point of loader.elf? Finally in case of firecracker which set VM to long mode and sets up paging the Linux way, what would be the proper way of resetting the VM state (paging) to OSv way? Switching back to 32-bit mode and calling start32? Waldek -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
