Re: Some Modest Virtualization Observations
On Tue, Mar 14, 2023 at 03:23:11PM -0300, Luiz Angelo Daros de Luca wrote: > > Yes, docs are quite outdated, mostly because everything now runs out > of the box. The x86/64 target does not have the embedded restrictions > in mind and the kernel has all VM drivers for most hypervisors > built-in (see: > https://github.com/openwrt/openwrt/blob/master/target/linux/x86/64/config-5.15). > It's just like any other modern appliance. Yet this loses some of the most important benefits of OpenWRT and VMs. The drivers for *all* hypervisors are built-in, instead of merely loading the appropriate ones as needed. Hence the recommendation of 256MB of memory, instead of the 128MB recommended for device purchases. Most hypervisor machines can handle this, but let the small VMs run loose and you'll be out of memory instantly. This fails as a developer simulcrum for an embedded device since too much memory is consumed to support all devices. Using modules for portions of the hypervisor support should cut down significantly. The kernel configurations also need work. Notice CONFIG_FB=y? For a router/AP you really don't need that feature. I suppose someone might need that in some circumstance, yet for production use that is major overhead. Speaking of support for all hypervisors. I notice "# CONFIG_XEN_BACKEND is not set". Most Xen device backends might be unnecessary, but CONFIG_XEN_NETDEV_BACKEND certainly seems worthwhile. > I run openwrt both in test and production using xen, qemu and > virtualbox and there is not much extra TODOs other than use the image > as a virtual disk, add some virtual NICs and have fun. The extra tips > also applies to any x86 target like "resize your image and its > partitions before flashing/upgrading if you need more room" or "a > custom grub, like from sles xen, might look for a config at a > different path (/boot/grub vs /boot/grub2)". Hyper V might work out of > the box, but I never used it. I'm trying to guess whether your network is rather complicated, versus "production" not being very stable for you. Using OpenWRT under all those for production at the same time seems odd. The real weakness in the documentation is it assumes a single virtual storage device. Whereas in a more advanced configuration one would have a read-only squashfs root in the VM. Then a separate virtual device for writeable data. There is no guidance for the how much space to allocate for writeable space for a x86 OpenWRT VM. -- (\___(\___(\__ --=> 8-) EHM <=-- __/)___/)___/) \BS (| ehem+sig...@m5p.com PGP 87145445 |) / \_CS\ | _ -O #include O- _ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445 ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Some Modest Virtualization Observations
> When reading https://openwrt.org/docs/guide-user/virtualization/start it > is clear this hasn't seen much love. Several portions seemed to > exclusively target developers and not really be meant for serious use. > As such I was unsurprised to discover: > https://openwrt.org/docs/guide-developer/start#testing_openwrt_in_a_virtual_machine > Which is exactly the same information in the developer guide. This area > seems to need work. > > One obvious problem is mixing Docker and LXC with virtualization. These > are useful for very basic testing, but they cannot handle many types of > testing which VMs can do. > > It is notable `xm` was deprecated by the Xen Project and was replaced > with `xl`. `xm` had completely disappeared by 2019 and was on its way > out by 2018. > > Both examples are also using older boot methods. If you're doing a new > setup, you likely want to use PvGRUB. PyGRUB is semi-deprecated (PvGRUB > is far superior on x86, not yet available on other arches) while direct > kernel boot is more often used to load PvGRUB or EDK2 as a bootloader. > > > > The image directly under "Where is my router?": > https://openwrt.org/docs/guide-user/virtualization/virtualbox-advanced#where_is_my_router > makes me wonder whether the creater of the image had an interesting > insight. > > A useful capability of modern hypervisors is the ability to pass hardware > devices (notably PCI and USB) into a VM. I've confirmed Xen, Bhyve and > KVM have this capability. I would be surprised if any other hypervisor > doesn't have the capability, if appropriate hardware is available > (notably an IOMMU). > > How exactly you do this will vary from hypervisor to hypervisor. The > basic idea though is the hypervisor removes/hides a device from the > supervisor VM, and installs it into another VM. This can be done with > most devices, notably graphics cards, ethernet NICs and 802.11 cards. > > The fun bit is, what happens if you have a hypervisor machine with a > spare ethernet NIC and 802.11 card, not being used for any other purpose? > On such a machine you could run OpenWRT in a VM. One NIC being the > physical ethernet interface. One NIC being the 802.11 interface. Then a > last interface going to the hypervisor's internal software switch. > > Thing is this has all the characteristics of a serious use of OpenWRT. > If the physical ethernet interface connects to your ISP, it can handle > NAT for both 802.11 devices and everything on the network. The only > difference being this merely uses /part/ of a high-powered computer, > rather than all of an embedded device. > > Certainly not my intended target, but imagining a Hyper-V system using > OpenWRT to handle the upstream connection seems an interesting idea. > > > The problems I forsee with this setup revolve around the choices made for > OpenWRT which are appropriate for embedded devices, but inappropriate for > dynamic systems. > > Notably much of OpenWRT is based on knowning more about the platform > ahead of time. Whereas when running on a hypervisor, most drivers should > be dynamically loaded. Except much of that support is stripped out by > OpenWRT to reduce kernel size. Yes, docs are quite outdated, mostly because everything now runs out of the box. The x86/64 target does not have the embedded restrictions in mind and the kernel has all VM drivers for most hypervisors built-in (see: https://github.com/openwrt/openwrt/blob/master/target/linux/x86/64/config-5.15). It's just like any other modern appliance. I run openwrt both in test and production using xen, qemu and virtualbox and there is not much extra TODOs other than use the image as a virtual disk, add some virtual NICs and have fun. The extra tips also applies to any x86 target like "resize your image and its partitions before flashing/upgrading if you need more room" or "a custom grub, like from sles xen, might look for a config at a different path (/boot/grub vs /boot/grub2)". Hyper V might work out of the box, but I never used it. The docs are in a wiki and you might be able to update it. It might have rotten because nobody really needs it anymore. However, if you looked for it and it didn't answer your questions, you are in the best position to add those extra questions and answers to the docs and delete outdated stuff. We are mostly volunteers that just scratch where it itches. Regards, Luiz ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Some Modest Virtualization Observations
When reading https://openwrt.org/docs/guide-user/virtualization/start it is clear this hasn't seen much love. Several portions seemed to exclusively target developers and not really be meant for serious use. As such I was unsurprised to discover: https://openwrt.org/docs/guide-developer/start#testing_openwrt_in_a_virtual_machine Which is exactly the same information in the developer guide. This area seems to need work. One obvious problem is mixing Docker and LXC with virtualization. These are useful for very basic testing, but they cannot handle many types of testing which VMs can do. It is notable `xm` was deprecated by the Xen Project and was replaced with `xl`. `xm` had completely disappeared by 2019 and was on its way out by 2018. Both examples are also using older boot methods. If you're doing a new setup, you likely want to use PvGRUB. PyGRUB is semi-deprecated (PvGRUB is far superior on x86, not yet available on other arches) while direct kernel boot is more often used to load PvGRUB or EDK2 as a bootloader. The image directly under "Where is my router?": https://openwrt.org/docs/guide-user/virtualization/virtualbox-advanced#where_is_my_router makes me wonder whether the creater of the image had an interesting insight. A useful capability of modern hypervisors is the ability to pass hardware devices (notably PCI and USB) into a VM. I've confirmed Xen, Bhyve and KVM have this capability. I would be surprised if any other hypervisor doesn't have the capability, if appropriate hardware is available (notably an IOMMU). How exactly you do this will vary from hypervisor to hypervisor. The basic idea though is the hypervisor removes/hides a device from the supervisor VM, and installs it into another VM. This can be done with most devices, notably graphics cards, ethernet NICs and 802.11 cards. The fun bit is, what happens if you have a hypervisor machine with a spare ethernet NIC and 802.11 card, not being used for any other purpose? On such a machine you could run OpenWRT in a VM. One NIC being the physical ethernet interface. One NIC being the 802.11 interface. Then a last interface going to the hypervisor's internal software switch. Thing is this has all the characteristics of a serious use of OpenWRT. If the physical ethernet interface connects to your ISP, it can handle NAT for both 802.11 devices and everything on the network. The only difference being this merely uses /part/ of a high-powered computer, rather than all of an embedded device. Certainly not my intended target, but imagining a Hyper-V system using OpenWRT to handle the upstream connection seems an interesting idea. The problems I forsee with this setup revolve around the choices made for OpenWRT which are appropriate for embedded devices, but inappropriate for dynamic systems. Notably much of OpenWRT is based on knowning more about the platform ahead of time. Whereas when running on a hypervisor, most drivers should be dynamically loaded. Except much of that support is stripped out by OpenWRT to reduce kernel size. -- (\___(\___(\__ --=> 8-) EHM <=-- __/)___/)___/) \BS (| ehem+sig...@m5p.com PGP 87145445 |) / \_CS\ | _ -O #include O- _ | / _/ 8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445 ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
