On Apr 7, 2018, at 19:04, Rainer Müller wrote:
> On 2018-04-07 19:44, Clemens Lang wrote:
>> For these resons, we want to reset the machines to a clean state before
>> every build, which we could do with buildbot, but requires some python
>> magic that hasn't been written yet.
> I think this is mostly written. What is needed is only a little bit of
> coding and a lot of testing, which is very time consuming.
> So here is the full plan in detail:

Thanks for writing this up.

> Buildbot supports libvirt with "latent buildslaves", even in 0.8.12:
> http://docs.buildbot.net/0.8.12/manual/cfg-buildslaves.html#libvirt
> When a build is about to be executed on this buildslave, Buildbot
> instructs clones the base image to a new (temporary) disk image and then
> uses this new disk image to spin up a VM from a clean state.
> To run this on ESX or any other hypervisor than qemu, we would need to
> implement the disk cloning manually as buildbot only uses qemu-img or cp
> on the local machine. Probably we would want to use snapshots instead of
> cloning full disks. We can customize this in parts by inheriting from
> LibVirtSlave.
> This VM will then boot and it is supposed to have a buildslave installed
> that will connect back to the buildbot master. From then on, the build
> runs as usual. After the build finished, buildbot will shutdown the VM
> and discard the modified disk image.
> The hardest part is to prepare this VM image. At best, it should be
> automated so we can easily recreate it for macOS updates.
> 1) Install macOS
>   Not sure how much can be automated here. For a basic version [1], it
>   can just be the instructions that need to be clicked through.
> 2) Install MacPorts into toolsprefix
> 3) Install buildslave (and other tools) into the toolsprefix
> 4) Install configuration for buildslave
>   Also set buildslave to start automatically after booting and to
>   connect to the master.
> 5) Install MacPorts into /opt/local

Yes there are tools to automate the OS install, like AutoDMG, which I have 
played with a few years ago, but there are many nuances and getting everything 
right is extremely time consuming, given how much time it takes to verify that 
any change had the desired outcome (since a complete OS install must be 
performed), not to mention how much disk space is required.

I don't see any reason to automate the creation of these VM images. It should 
be no more difficult to set them up manually than it was for me to set up the 
ones we already use for buildbot today.

> As far as I know, Ryan already has a script that could be used for some
> of the steps.

I have this:


Given a new machine with a fresh install of macOS, Xcode, and the command line 
tools, the script builds MacPorts from source in the private /opt/bblocal 
prefix used by the buildbot workers, installs the git port, and adds another 
ports tree from GitHub containing the ports used for the buildbot 
infrastructure. The administrator then installs the ports needed for this 
particular worker. (It's usually just the same port on each worker, except that 
they use different variants depending on whether the worker will build base or 
ports or both, and whether libc++ or libstdc++ will be used.)

> When we have such an, we need to configure a builder in master.cfg that
> uses LibVirtSlave. This builder needs a modified portbuilder_factory
> that skips the deployment of archives. And finally a changesource that
> listens to pull requests via a GitHub WebHook (or just some branch for
> testing at first).
> At this point the setup would already be working.
> If the goal is just to get anything working, you could stop reading
> here. But I imagine this setup to be slow, as the VM cannot keep any
> persistent state. Therefore, I already made up my mind how this could be
> optimized.
> As the buildbot has no local packages installed, operations like
> checking for the existence of a package over HTTP and then fetching the
> file will be slow. I expect this to be one major bottleneck. Our usual
> portbuilders have all previously built packages available locally already.
> One possible optimization would be to mount distfiles and packages in
> the VM over (read-only) NFS from another machine that provides the local
> mirror.

I'm not convinced that fetching via HTTP from the private mirror would be 
significantly slower than fetching via NFS from the private mirror. Either way, 
the files are coming from the fairly slow RAID card in the master Xserve.

> mount -t nfs -o ro mirror:.../allpackages \
>    /opt/local/var/macports/incoming/verified
> Now a 'port install' would just directly use the package from this
> directory, which is much faster than doing the usual archivefetch.
> However, this expects a flat hierarchy; a single directory containing
> all packages. This may impose limitations we have yet to find.

As you say, we don't have the packages in a flat hierarchy. How would you 
propose that we create that? Are we talking about yet another parallel 
hierarchy of hardlinks for our archive deployment script to maintain, like the 
one we've been talking about in another thread for rearranging the contents of 
the packages server?

> For distfiles, it would be possibile to apply a trick with a union mount
> to create a local overlay that allows writing new files:
> mount -t nfs -o ro,union mirror:.../distfiles \
>    /opt/local/var/macports/distfiles
> hdiutil create -o /tmp/distfiles-overlay.dmg -size 20g -fs HFS+J
> hdiutil attach -nomount /tmp/distfiles-overlay.dmg  # diskX in output
> mount -t hfs -o rw,union /dev/diskXs1 /opt/local/var/macports/distfiles
> With this, all file lookups are first attempted in the overlay and if
> the file does not exist there, it will be looked up in the layer below,
> which is our mirror. New files will always be written to the top layer.
> Therefore existing distfiles will be used from the mirror, but new
> distfiles can be written to the overlay.

Sure, I imagine that would be possible. But again, I'm not sure that fetching 
via NFS is faster than fetching from the same server via HTTP.

If we are thinking about having PR test machines running OS X 10.8 or earlier, 
then we have to consider the HTTPS fetch failure situation again. I don't think 
we want to mirror distfiles for unapproved PRs to the main distfiles server, so 
we may need a separate PR distfiles server that could house the files 
temporarily and delete them later.

> This email got a bit long, but apparently there is an interest in
> getting a detailed explanation of how this could be implemented. It is
> also good I finally took the time to write these things down, as I have
> been thinking about it for a while.
> Rainer
> [1] For an advanced version of automated macOS installation:
>  In the High Sierra installer, the startosinstall script has an
>  additional --installpackage option that can point to a .pkg to be
>  executed after the installation finished. If, and only if, you also
>  disable SIP, you can use the option --volume to install onto a
>  different disk than the one in your Mac. Maybe this could be combined
>  to fully automate all steps... but that is really advanced.

Reply via email to