On Sun, 4 Aug 2024 at 08:07, d <omn...@puptv.com> wrote:
> Is it possible, and how hard would it be to make a fork of the omicron1
> branded sparse VM for Helios that supports PkgSrc?
The source for the omicron1 brand is here:
https://github.com/oxidecomputer/helios-omicron-brand
It's not completely accurate to say that it's sparse, for what it's
worth. Sparse zones currently mount some trees, like /usr, read-only
from the host system. You then can't modify that tree at all; what's
in the GZ version of /usr is in the zone, no more, no less.
I wanted to add the ability to modify the contents of /usr, to be able
to add or remove files as part of layering on additional packages -- a
bit like a docker image does. We don't currently have a union file
system, and in fact union file systems are complex in (correct)
implementation, which is why I didn't just write one at the time.
Instead, I used some important facts about the intended structure of
Helios systems in the Oxide product (a ramdisk-based appliance) to
make a simpler approach.
At zone install time, we:
- create a blank ZFS file system
- unpack a baseline tar file to get a blank /etc and /var
- make a recursive _copy_ of whatever happens to be in /usr and
/lib in the GZ into the zone file system
- unpack more layer tar files on top of the result of the
previous steps, if provided
Both the base ramdisk image for the host, and the baseline tar file,
are assembled together from IPS packages at image build time. This
means they're never out of sync with one another. Also critically,
libc in the zone image is never out of date with the host kernel, even
though we made a copy, because it's all ramdisk ephemera anyway; when
we reboot and want to start more zones, we install them from scratch,
potentially mounting in delegated datasets that have the specific
persistent data we needed (like database files, etc).
This presents a challenge, though, for workstations and engineering
systems in the lab, where we use a classic install-to-disk UNIX system
approach that allows one to add packages and mutate the system in the
ways that are convenient in that setting. There, each time you "pkg
update" to get new OS software, the kernel _would_ be out of sync with
the libraries zone root. The baseline archive would also be out of
sync, and no package update actions would have occurred. We solve the
baseline issue by regenerating it in-situ using pkg(1) in a boot-time
SMF service after each boot.
As described in the omicron1(7) manual page, and the README, for
development systems it's your responsibility to hold it correctly.
This means any time you do a "pkg update" and reboot, you must also
uninstall and reinstall any zones after you reboot. This will use the
updated baseline, and will also re-copy all the files from /usr that
must be in sync. Ultimately, you must simulate what we do on the
ramdisk system for it all to hold together in the expected way.
I don't have a succinct answer to the question, but there are some
things that might help you answer it on your own:
- the pkgsrc bootstrap is just files to lay down in the file
system, so really you could probably just create a regular
omicron1 brand image layer tar file that you could pass
to the regular (unmodified) brand to get a zone that has
pkgsrc bits inside, just as we do with Omicron components
- the omicron1 brand is really, for production use, intended
to be used in a very particular way, in a very particular
ramdisk environment (described in the documentation and
above) -- if you aren't planning on building and booting
ramdisk images I'm not sure that I would use it, given
the caveats for its use on install-to-disk systems
- we made a conscious choice to focus on hardware-level
virtualisation in the Oxide product, in no small part due
to the (relative!) ease with which user workloads can
be punted between machines using live migration; this is
not something that's likely to be easy or even possible
to achieve with zone-based workloads -- if you're planning
to use our control plane, but on i86pc systems, I suspect
you're going to run into more and more friction as we
lean more heavily into the live migration stuff for
things like seamless software update and so on.
Cheers.
--
Joshua M. Clulow
http://blog.sysmgr.org
------------------------------------------
illumos: illumos-discuss
Permalink:
https://illumos.topicbox.com/groups/discuss/Te80aac52002d7fd2-M560aa7c2ef59a48cd441918d
Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
