On Fri, Mar 29, 2024 at 10:52:01AM -0700, Luis Chamberlain wrote: > + cc kdevops and Ross from the debian side. > > On Fri, Mar 29, 2024 at 09:17:52AM +0000, Richard W.M. Jones wrote: > > > On Thu, Mar 28, 2024 at 09:33:50AM -0700, Luis Chamberlain wrote: > > > > [ v2 as I don't think the v1 email got through to guestfs list ] > > > > > > > > Thanks to the work by Jeff Layton, we've embraced guestfs on kdevops > > > > [0] as a replacement to vagrant. To save bandwidth kdevops lets you > > > > set up mirrors for all git trees we need, it would be nice to make it > > > > easier for folks to also setup mirrors for guestfs base images, the > > > > base images are here: > > > > > > > > https://builder.libguestfs.org/ > > > > > > > > I take it we'd just add say > > > > $XDG_CONFIG_HOME/virt-builder/repos.d/kdevops-local-mirror.conf with > > > > the new URL where we established the mirror, is that correct? > > > > Yes. > > Neat! > > > > > I was looking to see if there was an rsync mirror for the main URL > > > > where all the images are above, or is there a better way to get the > > > > latest updates? > > > > Ideally we'd like to not host the images at all, but getting distros > > to host them instead has not been very successful (except for SUSE). > > It makes sense, it would put all the burden on just one server, and the > licensing considerations are already figured out by distros > as well so might as well leverage that. So I see four for distros > to host them: > > 1) decentralizes > 2) distros already produce cloud images, having a guestfs image > should be trivial to produce, just as current vagrant images > 3) licensing is already figured out > 4) since some distros also provie updated images than the base, > as with debian testing vagrant images, they can also provide > more up to date guestfs images as well, reducing the amount of > time it takes to have an up to date system. > > If distros do start shipping guestfs images, then eventually > we want to update upstream virt-builder with a respective: > > $XDG_CONFIG_HOME/virt-builder/repos.d/$distro.conf > > That would allo $distro's source spelled out for their own updates. > A good example as you hinted was that opensuse already does exactly > this, so we have: > > cat /etc/virt-builder/repos.d/opensuse.conf > [opensuse.org] > uri=http://download.opensuse.org/repositories/Virtualization:/virt-builder-images/images/index > gpgkey=file:///etc/virt-builder/repos.d/opensuse.gpg > > > > > Also, to save even more time with images, it would be useful if we had > > > > base images getting updates a bit more regularly so that the step to > > > > apt-get upgrade after bringup won't take too long by using the latest > > > > base images. It seems this is possible with customer images, but it > > > > would be nicer if we had this as a general thing. Any ideas to support > > > > that? > > > > Yes, we'd like Debian to generate their own images! Actually > > they don't even need to generate images at all, just host > > the index file. > > Great! Let's all start helping to persuade each distro to do this then. > > In the meantime, to help alleviate the load off of builder.libguestfs.org > it would be good to at laest host a few mirrors, maybe some public maybe > some not public. To help with this I'd like some clarifications. > > I just discovered and ran virt-builder --cache-all-templates, which > downloads all the 29 GiB of base images onto my ~/.cache/virt-builder/ > but I don't see the index file there, so this does not seem to be > an equivalent of what would be an rsync of the data on > https://builder.libguestfs.org/ so I don't think the contents of this > directory could be used without postprocessing of the same data on the > builder page.
So I might be missing something, but simply downloading the content of https://builder.libguestfs.org/ (eg using wget -r) will get everything you need including the index file. Mirroring the images is a complicated topic. Since the subject of supply chain attacks is foremost in my mind right now, we need to make sure that the chain of trust is maintained. Currently it is: - /etc/virt-builder/repos.d/*.conf containing a URL and GPG public key - https://builder.libguestfs.org/index.asc is GPG signed - index file contains sha512 checksums which match the individual images virt-builder should check all this, so as long as the index files and images aren't corrupted you can just point your own /etc/virt-builder/repos.d/mirror.conf at any mirror location you like. I don't have the ability to start hosting mirrors myself. > This is not an issue if all one wants is to just mirror > the data locally, but say if one wanted to *host* a exact mirror of > what you have in the URL it would need some post processing. For > instance the above URL has a XZ compressed debian-12.xz while the command > virt-builder --cache-all-templates gets me an XZ compressed file > ~/.cache/virt-builder/debian-12.x86_64.1 decompressing both and > sha1suming confirms its the same file with an sha1sum of > > 6fc19d34398c2c57ebd396c080d63531238f3ac43b61c66ae167903188404bf44de29b677085b9419f906a30cc720afc3fe43af7e09d970beadd8b673a06ca86 > > and with it compressed as reflected in the public index file: > > f1c58b53b7d25691ccbd2c3eb97f692985e8a035247dee3ed8e024bbae64574620ace75dbb57207064739f5e4b317c6ad6d6d518d777c719441b06b1634fe07e > > Is there a better way to host an exact copy of what's in > https://builder.libguestfs.org and what would a conf that > does this look like? I believe an experiment you can try is: (1) wget https://builder.libguestfs.org/index.asc and a few images to a hosting location of your choice (2) change /etc/virt-builder/repos.d/libguestfs.conf to point to the new location (3) virt-builder should function as normal (if you delete the local cache, it should download again) (4) if you modify index.asc or one of the images, virt-builder should reject it > Let's say we want to implement private or public mirrors to help, > once we figure out how to mirror mirror same data, and we dump it on > page say on foobar.org would we want to add to a system: > > /etc/virt-builder/repos.d/foobar.conf > > [foobar.org] > uri=http://foobar.org/download/builder/index.asc > gpgkey=file:///etc/xdg/virt-builder/repos.d/libguestfs.gpg Yes. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html _______________________________________________ Libguestfs mailing list -- guestfs@lists.libguestfs.org To unsubscribe send an email to guestfs-le...@lists.libguestfs.org
