Colin Watson has proposed merging ~cjwatson/launchpad:doc-services-build-farm into launchpad:master.
Commit message: doc: Add high-level build farm documentation Requested reviews: Launchpad code reviewers (launchpad-reviewers) For more details, see: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/452655 Written collaboratively during a hackathon. -- Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:doc-services-build-farm into launchpad:master.
diff --git a/doc/reference/index.rst b/doc/reference/index.rst index 0fd2897..836b6f2 100644 --- a/doc/reference/index.rst +++ b/doc/reference/index.rst @@ -15,6 +15,7 @@ Services .. toctree:: :maxdepth: 1 + services/build-farm services/signing Possibly out-of-date diff --git a/doc/reference/services/build-farm.rst b/doc/reference/services/build-farm.rst new file mode 100644 index 0000000..5dccb6b --- /dev/null +++ b/doc/reference/services/build-farm.rst @@ -0,0 +1,164 @@ +Build farm +========== + +Short description +----------------- +Builds and tests software for Ubuntu and related systems. + +Detailed description +-------------------- +The build farm builds packages, such as debs, snaps, charms, and other +formats for Ubuntu in a secure and isolated way. + +The build farm consists of a manager, a proxy for outbound communication, +and various builder regions. +Build jobs are created by other parts of Launchpad, such as a change to a +git repository resulting in snap recipe builds. +They are then scheduled onto builders as virtual machines by +``buildd-manager``. + +``buildd-manager`` continuously monitors all builders in the farm, and when +a builder is idle it chooses the next highest priority job for dispatch to +the idle builder. +It does this by using XML-RPC, passing relevant data to the builder, which +in turn fetches all ingredients for executing the build. +When a build has completed, ``buildd-manager`` gathers the resulting files +and injects them into Launchpad via an upload queue processor, and then the +builder is reset to a baseline state. + +The manner in which a build is executed is determined by the build type. +This typically involves invoking some external tools; ``sbuild`` for Ubuntu +package builds, ``snapcraft`` for snap builds, etc. + +Builders do not have direct access to the internet, but rather need to +acquire an authentication token to be able to access a restricted set of +URLs on the internet via a squid proxy. + +Builder regions are physically co-located and consist of machines of the +same architecture family. +Builder regions comprise per-architecture image builders, and each physical +location contains a ``launchpad-vbuilder-manage`` instance which handles +resets for all architectures within that location. + +Each builder region maintains clean VM images for its builders; these are +built using ``glance-simplestreams-sync``, which automatically and +periodically copies standard pre-built images from +``cloud-images.ubuntu.com``, and ``launchpad-buildd-image-modifier``, which +hooks into ``glance-simplestreams-sync`` to produce modified images with +``launchpad-buildd`` installed. + +Documentation +------------- +* `Documentation for launchpad-buildd <https://launchpad-buildd.readthedocs.io/en/latest/index.html>`_ +* `Documentation for the Launchpad CI runner <https://lpci.readthedocs.io/en/latest/>`_ +* `Documentation for Launchpad CI <https://help.launchpad.net/Code/ContinuousIntegration>`_ + +Git repositories +---------------- +* `buildd-manager <https://git.launchpad.net/launchpad/tree/lib/lp/buildmaster/>`_ +* `launchpad-buildd <https://git.launchpad.net/launchpad-buildd>`_ +* `Launchpad CI runner <https://git.launchpad.net/lpci>`_ +* `glance-simplestreams-sync charm <https://git.launchpad.net/~launchpad/charm-glance-simplestreams-sync/tree/?h=scalingstack>`_ +* `launchpad-buildd-image-modifier charm <https://git.launchpad.net/charm-launchpad-buildd-image-modifier>`_ +* `launchpad-vbuilder-manage <https://git.launchpad.net/launchpad-vbuilder-manage>`_ + +Bug trackers +------------ +* https://bugs.launchpad.net/launchpad-project/+bugs?field.tag=soyuz-build +* https://bugs.launchpad.net/launchpad-buildd +* https://bugs.launchpad.net/lpci +* https://bugs.launchpad.net/charm-glance-simplestreams-sync +* https://bugs.launchpad.net/charm-launchpad-buildd-image-modifier +* https://bugs.launchpad.net/launchpad-vbuilder-manage + +Deployment +---------- +* `Deployment of launchpad-buildd <https://launchpad-buildd.readthedocs.io/en/latest/how-to/deployment.html>`_ +* `Production deployment notes for launchpad-buildd <https://launchpad-buildd.readthedocs.io/en/latest/explanation/deployment.html>`_ +* `vbuilder Mojo spec <https://git.launchpad.net/~launchpad/launchpad-mojo-specs/+git/private/tree/vbuilder?h=vbuilder>`_ +* `lp-builder-proxy Mojo spec <https://git.launchpad.net/launchpad-mojo-specs/tree/lp-builder-proxy/>`_ + +Related specifications +---------------------- +`LP113: Threat model for builds <https://docs.google.com/document/d/1im8CMxLRNxtt5H0zv461kSYSflN-YlxJ1UZG8_53D9A>`_ + +Log files +--------- +See https://wiki.canonical.com/Launchpad/FreshLogs. + +Production +~~~~~~~~~~ + +buildd-manager +^^^^^^^^^^^^^^ + +* ``rless buildd-manager.lp.internal::lp-logs/buildd-manager.log`` +* ``rless buildd-manager.lp.internal::lp-logs/process-build-uploads.log`` +* ``rless buildd-manager.lp.internal::lp-logs/buildd-retry-depwait.log`` + +vbuilder-manage +^^^^^^^^^^^^^^^ + +Builder reset logs. +Each celery worker has a different log file, named ``celery.ppareset-*.log``. + +* ``rsync -v vbuilder-manage-lcy02.lp.internal::vbuilder-manage-logs/`` +* ``rsync -v vbuilder-manage-bos01.lp.internal::vbuilder-manage-logs/`` +* ``rsync -v vbuilder-manage-bos02.lp.internal::vbuilder-manage-logs/`` +* ``rsync -v vbuilder-manage-bos03.lp.internal::vbuilder-manage-logs/`` + +builder-proxy (auth) +^^^^^^^^^^^^^^^^^^^^ + +* ``rless builder-proxy-auth.lp.internal::rutabaga-logs/rutabaga-access.log`` +* ``rless builder-proxy-auth.lp.internal::rutabaga-logs/rutabaga-error.log`` +* ``rless builder-proxy-auth.lp.internal::rutabaga-logs/rutabaga-purge.log`` + +builder-proxy (squid) +^^^^^^^^^^^^^^^^^^^^^ + +* ``rless 10.131.48.38::squid-logs/access.log`` +* ``rless 10.131.48.38::squid-logs/cache.log`` +* ``rless 10.131.48.24::squid-logs/access.log`` +* ``rless 10.131.48.24::squid-logs/cache.log`` + +Staging +~~~~~~~ + +buildd-manager +^^^^^^^^^^^^^^ + +* ``rless 10.132.54.143::lp-logs/buildd-manager.log`` +* ``rless 10.132.54.143::lp-logs/process-build-uploads.log`` +* ``rless 10.132.54.143::lp-logs/buildd-retry-depwait.log`` + +builder-proxy (auth) +^^^^^^^^^^^^^^^^^^^^ + +* ``rless builder-proxy-auth.staging.lp.internal::rutabaga-logs/rutabaga-access.log`` +* ``rless builder-proxy-auth.staging.lp.internal::rutabaga-logs/rutabaga-error.log`` +* ``rless builder-proxy-auth.staging.lp.internal::rutabaga-logs/rutabaga-purge.log`` + +builder-proxy (squid) +^^^^^^^^^^^^^^^^^^^^^ + +* ``rless 10.132.224.179::squid-logs/access.log`` +* ``rless 10.132.224.179::squid-logs/cache.log`` +* ``rless 10.132.224.16::squid-logs/access.log`` +* ``rless 10.132.224.16::squid-logs/cache.log`` + +Monitoring +---------- +The "Build farm" section of the `Launchpad dash <https://grafana.admin.canonical.com/d/oIhMaXhMk/launchpad-dash>`_. + +Search for "build" in https://git.launchpad.net/canonical-is-prometheus/tree/ols/launchpad.rules. + +Common support cases +-------------------- +See `Launchpad's playbook for support rotation <https://wiki.canonical.com/Launchpad/SupportRotation#Builder>`_ + +More information +---------------- + +* `Live overview of Launchpad's build farm <https://launchpad.net/builders>`_ +* `Launchpad services diagram <https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&edit=_blank&layers=1&nav=1&page-id=14glVH8XSJX-2FxTRWny#G1j-yk3c4mzYfMC79Y-uo9__u93pLWkiMi>`_
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : launchpad-reviewers@lists.launchpad.net Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp