Re: [osv-dev] Re: [RFC] Remaining work to support OSv on firecracker

[Waldek Kozaczuk]

On Monday, February 11, 2019 at 3:54:39 AM UTC-5, Nadav Har'El wrote:
>
> On Sun, Feb 10, 2019 at 10:50 PM Waldek Kozaczuk  > wrote:
>
>> This patch is not meant to be applied but it serves to illustrate what is 
>> left to make OSv boot on firecracker. As you can see it is not a daunting 
>> amount of work anymore given modern virtio is supported now.
>>
>> In am not sure in what order we should implement the remaining work. 
>> Ideally every patch should be testable
>>
>
> Indeed, although I don't mind that the firecracker-specific stuff you'll 
> be the only one testing it (since I'm not planning to run it any time soon).
>
> Maybe you can even add to run.py a firecracker hypersor options, so if 
> someone just installs this hypervisor he can use run.py?
>
It is definitely doable. 

>
>  
>
>> which means that virtio-mmio support should be added last given it is not 
>> possible to test it on QEMU (this 
>> https://github.com/qemu/qemu/blob/595123df1d54ed8fbab9e1a73d5a58c5bb71058f/hw/virtio/virtio-mmio.
>>  
>> 
>>
> seems to suggest that QEMU is able to emulate virtio-mmio devices but I 
>> have not found any examples of using it on x86-64 platform, anyone has 
>> ideas?). Please see my extra comments/questions below.
>>
>> On Sunday, February 10, 2019 at 3:22:44 PM UTC-5, Waldek Kozaczuk wrote:
>>>
>>> - clean up modern/legacy virtio support (configurability) 
>>>
>> This mostly is about fragments of code in virtio layer similar to this - 
>> https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-blk.cc#L177-L187
>>  
>> - where code needs to be tweaked to properly identify from feature flags 
>> what configuration data is available and only read this.  
>>
>>> - add virtio-mmio support including parsing virtio-mmio devices from 
>>> command line 
>>> - enhance boot.S and OSv boot in general to support vmlinux format 
>>> (decompressed ELF 64-bit Linux kernel) 
>>>- pass e820 and command line parameters through zero-page 
>>>
>>  This is most non-trivial amount of work that requires changing OSv boot 
>> protocol so it better conforms to a Linux (for reference 
>> https://www.kernel.org/doc/Documentation/x86/boot.txt). It would be nice 
>> to support both 32-bit and 64-bit ELF direct boot LInux protocol. It is not 
>> clear to me what is the best way to support it especially if we still want 
>> to be able to boot OSv in real mode, protected and long mode at the same 
>> time and ideally minimize number of versions of boot artifacts (usr.img, 
>> lzloader.elf, loader.elf, etc). Maybe this deserves its own email thread.
>>
> Do you have any ideas around supporting 32-bit and 64-bit ELF direct boot 
LInux protocol? Ideally I would would want to avoid extra loader ELF 
version but single ELF cannot have 2 entry points - one for 32-bit and 
another for 64-bit, can it?

> - make APCI optional and/or disable-able which includes: 
>>>- make pvpanic probe only if ACPI on 
>>>- make poweroff work when ACPI off 
>>>- parse SMP information from MP table instead of MADT if ACPI off 
>>>
>>
>> I think that all the 3 ACPI related items above can be part of a single 
>> patch. 
>>
>>> - add option to turn PCI off (make OSv not even try to enumerate PCI) 
>>> - mark boot time end 
>>
>>
>>> Signed-off-by: Waldemar Kozaczuk > 
>>> --- 
>>>  Makefile   |   1 + 
>>>  arch/x64/arch-setup.cc | 180 - 
>>>  arch/x64/boot.S|  37 - 
>>>  arch/x64/loader.ld |   2 +- 
>>>  arch/x64/power.cc  |  14 +--- 
>>>  arch/x64/smp.cc|  16 +++- 
>>>  drivers/acpi.cc|  10 ++- 
>>>  drivers/virtio-blk.cc  |   8 +- 
>>>  drivers/virtio-mmio.cc | 131 ++ 
>>>  drivers/virtio-mmio.hh | 148 + 
>>>  drivers/virtio-net.cc  |   7 ++ 
>>>  drivers/virtio.cc  |   7 +- 
>>>  drivers/virtio.hh  |   1 + 
>>>  loader.cc  |   5 +- 
>>>  14 files changed, 527 insertions(+), 40 deletions(-) 
>>>  create mode 100644 drivers/virtio-mmio.cc 
>>>  create mode 100644 drivers/virtio-mmio.hh 
>>>
>>> diff --git a/Makefile b/Makefile 
>>> index ff9f2ed3..b905063c 100644 
>>> --- a/Makefile 
>>> +++ b/Makefile 
>>> @@ -821,6 +821,7 @@ drivers += arch/$(arch)/pvclock-abi.o 
>>>  drivers += drivers/virtio.o 
>>>  drivers += drivers/virtio-pci-device.o 
>>>  drivers += drivers/virtio-vring.o 
>>> +drivers += drivers/virtio-mmio.o 
>>>  drivers += drivers/virtio-net.o 
>>>  drivers += drivers/vmxnet3.o 
>>>  drivers += drivers/vmxnet3-queues.o 
>>> diff --git a/arch/x64/arch-setup.cc b/arch/x64/arch-setup.cc 
>>> index 50500662..cf5adb15 100644 
>>> --- a/arch/x64/arch-setup.cc 
>>> +++ b/arch/x64/arch-setup.cc 
>>> @@ -22,6 +22,19 @@ 
>>>  #include  
>>>  #include "dmi.hh" 
>>>   
>>> +// Not sure if Linux zero page is always located at this place 
>>> +// in memory or 

Re: [osv-dev] Re: [RFC] Remaining work to support OSv on firecracker

[Pekka Enberg]

On Mon, Feb 11, 2019 at 10:54 AM Nadav Har'El  wrote:

> Maybe you can even add to run.py a firecracker hypersor options, so if
> someone just installs this hypervisor he can use run.py?
>

That's a very good idea!

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Re: [RFC] Remaining work to support OSv on firecracker

[Nadav Har'El]

On Sun, Feb 10, 2019 at 10:50 PM Waldek Kozaczuk 
wrote:

> This patch is not meant to be applied but it serves to illustrate what is
> left to make OSv boot on firecracker. As you can see it is not a daunting
> amount of work anymore given modern virtio is supported now.
>
> In am not sure in what order we should implement the remaining work.
> Ideally every patch should be testable
>

Indeed, although I don't mind that the firecracker-specific stuff you'll be
the only one testing it (since I'm not planning to run it any time soon).

Maybe you can even add to run.py a firecracker hypersor options, so if
someone just installs this hypervisor he can use run.py?



> which means that virtio-mmio support should be added last given it is not
> possible to test it on QEMU (this
> https://github.com/qemu/qemu/blob/595123df1d54ed8fbab9e1a73d5a58c5bb71058f/hw/virtio/virtio-mmio.
> 
>
seems to suggest that QEMU is able to emulate virtio-mmio devices but I
> have not found any examples of using it on x86-64 platform, anyone has
> ideas?). Please see my extra comments/questions below.
>
> On Sunday, February 10, 2019 at 3:22:44 PM UTC-5, Waldek Kozaczuk wrote:
>>
>> - clean up modern/legacy virtio support (configurability)
>>
> This mostly is about fragments of code in virtio layer similar to this -
> https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-blk.cc#L177-L187
> - where code needs to be tweaked to properly identify from feature flags
> what configuration data is available and only read this.
>
>> - add virtio-mmio support including parsing virtio-mmio devices from
>> command line
>> - enhance boot.S and OSv boot in general to support vmlinux format
>> (decompressed ELF 64-bit Linux kernel)
>>- pass e820 and command line parameters through zero-page
>>
>  This is most non-trivial amount of work that requires changing OSv boot
> protocol so it better conforms to a Linux (for reference
> https://www.kernel.org/doc/Documentation/x86/boot.txt). It would be nice
> to support both 32-bit and 64-bit ELF direct boot LInux protocol. It is not
> clear to me what is the best way to support it especially if we still want
> to be able to boot OSv in real mode, protected and long mode at the same
> time and ideally minimize number of versions of boot artifacts (usr.img,
> lzloader.elf, loader.elf, etc). Maybe this deserves its own email thread.
>
>> - make APCI optional and/or disable-able which includes:
>>- make pvpanic probe only if ACPI on
>>- make poweroff work when ACPI off
>>- parse SMP information from MP table instead of MADT if ACPI off
>>
>
> I think that all the 3 ACPI related items above can be part of a single
> patch.
>
>> - add option to turn PCI off (make OSv not even try to enumerate PCI)
>> - mark boot time end
>
>
>> Signed-off-by: Waldemar Kozaczuk 
>> ---
>>  Makefile   |   1 +
>>  arch/x64/arch-setup.cc | 180 -
>>  arch/x64/boot.S|  37 -
>>  arch/x64/loader.ld |   2 +-
>>  arch/x64/power.cc  |  14 +---
>>  arch/x64/smp.cc|  16 +++-
>>  drivers/acpi.cc|  10 ++-
>>  drivers/virtio-blk.cc  |   8 +-
>>  drivers/virtio-mmio.cc | 131 ++
>>  drivers/virtio-mmio.hh | 148 +
>>  drivers/virtio-net.cc  |   7 ++
>>  drivers/virtio.cc  |   7 +-
>>  drivers/virtio.hh  |   1 +
>>  loader.cc  |   5 +-
>>  14 files changed, 527 insertions(+), 40 deletions(-)
>>  create mode 100644 drivers/virtio-mmio.cc
>>  create mode 100644 drivers/virtio-mmio.hh
>>
>> diff --git a/Makefile b/Makefile
>> index ff9f2ed3..b905063c 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -821,6 +821,7 @@ drivers += arch/$(arch)/pvclock-abi.o
>>  drivers += drivers/virtio.o
>>  drivers += drivers/virtio-pci-device.o
>>  drivers += drivers/virtio-vring.o
>> +drivers += drivers/virtio-mmio.o
>>  drivers += drivers/virtio-net.o
>>  drivers += drivers/vmxnet3.o
>>  drivers += drivers/vmxnet3-queues.o
>> diff --git a/arch/x64/arch-setup.cc b/arch/x64/arch-setup.cc
>> index 50500662..cf5adb15 100644
>> --- a/arch/x64/arch-setup.cc
>> +++ b/arch/x64/arch-setup.cc
>> @@ -22,6 +22,19 @@
>>  #include 
>>  #include "dmi.hh"
>>
>> +// Not sure if Linux zero page is always located at this place
>> +// in memory or its address is passed in one of the registers
>> +// -> double check
>> +#define ZERO_PAGE_START  0x7000
>> +#define SETUP_HEADER_OFFSET  0x1f1   // look at bootparam.h in linux
>> +#define BOOT_FLAG_OFFSET sizeof(u8) + 4 * sizeof(u16) + sizeof(u32)
>> +
>> +#define E820_ENTRIES_OFFSET  0x1e8   // look at bootparam.h in linux
>> +#define E820_TABLE_OFFSET0x2d0   // look at bootparam.h in linux
>> +
>> +#define CMD_LINE_PTR_OFFSET  sizeof(u8) * 5 + sizeof(u16) * 11 +
>> sizeof(u32) * 7
>> +#define CMD_LINE_SIZE_OFFSET CMD_LINE_PTR_OFFSET 

[osv-dev] Re: [RFC] Remaining work to support OSv on firecracker

[Waldek Kozaczuk]

This patch is not meant to be applied but it serves to illustrate what is 
left to make OSv boot on firecracker. As you can see it is not a daunting 
amount of work anymore given modern virtio is supported now.

In am not sure in what order we should implement the remaining work. 
Ideally every patch should be testable which means that virtio-mmio support 
should be added last given it is not possible to test it on QEMU (this 
https://github.com/qemu/qemu/blob/595123df1d54ed8fbab9e1a73d5a58c5bb71058f/hw/virtio/virtio-mmio.c
 
seems to suggest that QEMU is able to emulate virtio-mmio devices but I 
have not found any examples of using it on x86-64 platform, anyone has 
ideas?). Please see my extra comments/questions below.

On Sunday, February 10, 2019 at 3:22:44 PM UTC-5, Waldek Kozaczuk wrote:
>
> - clean up modern/legacy virtio support (configurability) 
>
This mostly is about fragments of code in virtio layer similar to this 
- 
https://github.com/cloudius-systems/osv/blob/master/drivers/virtio-blk.cc#L177-L187
 
- where code needs to be tweaked to properly identify from feature flags 
what configuration data is available and only read this.  

> - add virtio-mmio support including parsing virtio-mmio devices from 
> command line 
> - enhance boot.S and OSv boot in general to support vmlinux format 
> (decompressed ELF 64-bit Linux kernel) 
>- pass e820 and command line parameters through zero-page 
>
 This is most non-trivial amount of work that requires changing OSv boot 
protocol so it better conforms to a Linux (for reference 
https://www.kernel.org/doc/Documentation/x86/boot.txt). It would be nice to 
support both 32-bit and 64-bit ELF direct boot LInux protocol. It is not 
clear to me what is the best way to support it especially if we still want 
to be able to boot OSv in real mode, protected and long mode at the same 
time and ideally minimize number of versions of boot artifacts (usr.img, 
lzloader.elf, loader.elf, etc). Maybe this deserves its own email thread.

> - make APCI optional and/or disable-able which includes: 
>- make pvpanic probe only if ACPI on 
>- make poweroff work when ACPI off 
>- parse SMP information from MP table instead of MADT if ACPI off 
>

I think that all the 3 ACPI related items above can be part of a single 
patch. 

> - add option to turn PCI off (make OSv not even try to enumerate PCI) 
> - mark boot time end 


> Signed-off-by: Waldemar Kozaczuk  
> --- 
>  Makefile   |   1 + 
>  arch/x64/arch-setup.cc | 180 - 
>  arch/x64/boot.S|  37 - 
>  arch/x64/loader.ld |   2 +- 
>  arch/x64/power.cc  |  14 +--- 
>  arch/x64/smp.cc|  16 +++- 
>  drivers/acpi.cc|  10 ++- 
>  drivers/virtio-blk.cc  |   8 +- 
>  drivers/virtio-mmio.cc | 131 ++ 
>  drivers/virtio-mmio.hh | 148 + 
>  drivers/virtio-net.cc  |   7 ++ 
>  drivers/virtio.cc  |   7 +- 
>  drivers/virtio.hh  |   1 + 
>  loader.cc  |   5 +- 
>  14 files changed, 527 insertions(+), 40 deletions(-) 
>  create mode 100644 drivers/virtio-mmio.cc 
>  create mode 100644 drivers/virtio-mmio.hh 
>
> diff --git a/Makefile b/Makefile 
> index ff9f2ed3..b905063c 100644 
> --- a/Makefile 
> +++ b/Makefile 
> @@ -821,6 +821,7 @@ drivers += arch/$(arch)/pvclock-abi.o 
>  drivers += drivers/virtio.o 
>  drivers += drivers/virtio-pci-device.o 
>  drivers += drivers/virtio-vring.o 
> +drivers += drivers/virtio-mmio.o 
>  drivers += drivers/virtio-net.o 
>  drivers += drivers/vmxnet3.o 
>  drivers += drivers/vmxnet3-queues.o 
> diff --git a/arch/x64/arch-setup.cc b/arch/x64/arch-setup.cc 
> index 50500662..cf5adb15 100644 
> --- a/arch/x64/arch-setup.cc 
> +++ b/arch/x64/arch-setup.cc 
> @@ -22,6 +22,19 @@ 
>  #include  
>  #include "dmi.hh" 
>   
> +// Not sure if Linux zero page is always located at this place 
> +// in memory or its address is passed in one of the registers 
> +// -> double check 
> +#define ZERO_PAGE_START  0x7000 
> +#define SETUP_HEADER_OFFSET  0x1f1   // look at bootparam.h in linux 
> +#define BOOT_FLAG_OFFSET sizeof(u8) + 4 * sizeof(u16) + sizeof(u32) 
> + 
> +#define E820_ENTRIES_OFFSET  0x1e8   // look at bootparam.h in linux 
> +#define E820_TABLE_OFFSET0x2d0   // look at bootparam.h in linux 
> + 
> +#define CMD_LINE_PTR_OFFSET  sizeof(u8) * 5 + sizeof(u16) * 11 + 
> sizeof(u32) * 7 
> +#define CMD_LINE_SIZE_OFFSET CMD_LINE_PTR_OFFSET + sizeof(u8) * 2 + 
> sizeof(u16) + sizeof(u32) * 3 
> + 
>  struct multiboot_info_type { 
>  u32 flags; 
>  u32 mem_lower; 
> @@ -61,12 +74,81 @@ struct e820ent { 
>  u32 type; 
>  } __attribute__((packed)); 
>   
> +struct _e820ent { 
> +u64 addr; 
> +u64 size; 
> +u32 type; 
> +} __attribute__((packed)); 
> + 
>  osv_multiboot_info_type* osv_multiboot_info; 
>   
> -void parse_cmdline(multiboot_info_type& mb) 
> +struct mmio_device_info { 
> +u64