William Grant has proposed merging lp:~wgrant/launchpad/remove-obsolete-buildd-junk into lp:launchpad/devel.
Requested reviews: Launchpad code reviewers (launchpad-reviewers) This branch removes obsolete junk from launchpad-buildd. Deleted are some ancient preliminary build farm design docs and example.chroot, an example configuration file for buildd-slave-chroot-tool, which was itself removed some time ago. -- https://code.launchpad.net/~wgrant/launchpad/remove-obsolete-buildd-junk/+merge/30233 Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/remove-obsolete-buildd-junk into lp:launchpad/devel.
=== modified file 'lib/canonical/buildd/debian/rules' --- lib/canonical/buildd/debian/rules 2010-06-12 03:48:34 +0000 +++ lib/canonical/buildd/debian/rules 2010-07-19 04:56:00 +0000 @@ -60,7 +60,6 @@ install -m755 buildd-config.py $(target)/usr/bin/buildd-genconfig install -m644 sbuildrc $(targetshare)/sbuildrc install -m644 sbuild.conf $(target)/etc/sbuild.conf - install -m644 example.chroot debian/launchpad-buildd/usr/share/doc/launchpad-buildd/ install -m755 debian/upgrade-config $(targetshare)/upgrade-config install -m755 check-implicit-pointer-functions $(target)/usr/bin/check-implicit-pointer-functions # Okay, that's installed all the slave-related files === removed file 'lib/canonical/buildd/example.chroot' --- lib/canonical/buildd/example.chroot 2009-12-16 00:16:24 +0000 +++ lib/canonical/buildd/example.chroot 1970-01-01 00:00:00 +0000 @@ -1,46 +0,0 @@ -# Example chroot configuration file - -[chroot] -# What distroarchrelease are we after -distribution=ubuntu -distroseries=breezy -# Specify "host" to build for this host, otherwise try specifying an archtag -architecture=host - -# We usually want the buildd variant -variant=buildd - -# Where does debootstrap find the archive? -debootstraparchive=http://archive.ubuntu.com/ubuntu - -[apt] -# What is the archive root? -archiveroot=http://dogfood.internal/ubuntu -# What components do we enable? -components=main restricted universe -# What pockets (other than the release pocket?) -#pockets=security updates -pockets= - -# Disabling the verification is handy because it prevents issues -# where the signing key changes or the signatures are not available -signingdisabled=1 - -[buildd] -# Information for setting up the buildd configuration inside the chroot -user=buildd -group=buildd -paths=/build /build/buildd /home/buildd /var/debbuild /var/debbuild/srcdep-lock -touchfiles=/var/debbuild/avg-build-times /var/debbuild/avg-build-space /CurrentlyBuilding - -# Extra packages we should install for the buildd -extrapkgs=build-essential fakeroot - -[intervention] -# At what stages would you like to intervene in the process? -postbootstrap=0 -postextrapkgs=0 -postfinalapt=0 -# Currently this is directly after postfinalapt -pretarup=1 - === removed file 'lib/canonical/buildd/notes.txt' --- lib/canonical/buildd/notes.txt 2007-10-31 19:03:15 +0000 +++ lib/canonical/buildd/notes.txt 1970-01-01 00:00:00 +0000 @@ -1,206 +0,0 @@ -Launchpad Build Infrastructure -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Points of concern currently... - -SourcePackageRelease records may not be associated with any specific -DistroSeries. If that happens to be the case; what DistroArchSeries -do we build them for? - - -Database changes -^^^^^^^^^^^^^^^^ - -CREATE TABLE BuildQueue ( - id SERIAL PRIMARY KEY, - build INTEGER FOREIGN KEY REFERENCES build(id) NOT NULL, - builder INTEGER FOREIGN KEY REFERENCES builder(id), - created TIMESTAMP WITH TIMEZONE NOT NULL, - buildstart TIMESTAMP WITH TIMEZONE, - logtail TEXT, - -); - --- If speedindex is NULL, we don't know, else it is the number of seconds --- needed to build a reference package or something. -ALTER TABLE Builder ADD COLUMN speedindex INTEGER; - -Basic case decision on what to build -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For each DistroSeries which is currently marked for -autobuilding... Find all SourcePackageReleases published into that -DistroSeries for which the set of Build records does not contain -entirely the set of DistroArchSeries being built for that -DistroSeries. - -For each of those; create a build record in "Needs Building" state - -Next gather every Build record in the "Needs Building" state for which -no BuildQueue entry exists and create the BuildQueue entry. - -For each entry in the BuildQueue for whom the Build record states a -builder, query the builder for that build's state. - - * complete successful builds are extracted; recorded; the BuildQueue - entry removed. The time to build is recorded for future estimates. - * failed builds are logged as such, the Build record is marked as - "Failed to build" and the BuildQueue entry is removed. - * failed builds (failing due to dependencies) have their logs recorded - and the BuildQueue.builder is cleared; the Build is placed into - "Manual Dep Wait" state - * failed builds (failing due to chroot issues) have the same - done for them and are placed into "Chroot Wait" state. Also a - chroot-damage mail is sent out. - * failed builds (failing due to disk issues) are simply returned to - the "Needs Building" state, the builder is marked as failed and - a builder-failure mail is sent out. If that was the last builder - for a given architecture then lots of panicy emails are sent. - * incomplete builds have their logtails updated. - -For each entry in the BuildQueue for whom the Build record states -"Needs Building" (rather than ex. "Manual Dep Wait") - - * Calculate for each entry the chroot id needed. - * Any entry for which the chroot is unavailable is placed into - "Chroot Wait" state and is pared from the list - * Any entry for which the chroot is known bad is placed into the - "Chroot Wait" state - * Any entry for which there is currently no available builder is - pared from the list. - * For each remaining entry; the build-dependencies are analysed out - and a decision is made about whether or not the build is likely - to dep-wait or not. Any for which a dep-wait is inevitable is pared - out and left as "Needs Building" - * For each remaining entry a score is calculated. This is based upon - the score generated by the Distro{,Release}'s policy engine for - deciding what order things get built. Also taken into account are - things like whether or not the sourcepackage has been built on that - DistroSeries before. - Also an estimation of the time to build the package is calculated. - This is measured as a fraction/multiple of the time it takes to - build a "standard" package. This combined with the speed measure - of each builder available for the architecture provides us with - an estimate of how long the package will take to build on any given - builder. - * For each architecture in turn, order the pending builds by their - score. For each builder except the last; choose the next highest - scoring package and queue the start of the build on the builder. - For the last builder; reorder by estimated time to build (in an - exponential growth grouping) and by score. Queue the highest scoring - package in the shortest-time-to-build group. - * For each queued action; download the required chroot to the builder - if it doesn't already have it; download the sources to it and then - initiate the build. Record the fact that it is building. - -The scoring mechanism for pending builds -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - * Estimated timeto run (smaller == more points) - * Time spent in the build queue (longer == more points) - * Position in build dependency chains (depended on == more points) - * Distribution specific policy (E.g. toolchains first) - * Priority of package (Base before Desktop before Supported) - * Urgency of upload - * Overarching policy (E.g. security builds first) - -Time groupings for packages -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - * Five minutes or less - * Five to fifteen minutes - * Fifteen to thirty minutes - * Thirty to sixty minutes - * Greater than sixty minutes - - -The build-master <-> build slave protocol -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Operated entirely over XML-RPC... Always initiated by the master to -the slave; the only outgoing sockets the slave is permitted are direct -to the public archive machine. - -Buildd slaves have HTTP read-only access to librarian. - -Buildd master is called Fiera -Buildd slaves are called Bob - -Calls that the master can make to the slave -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - - INFO (none) -> - Slave Protocol Version - [ Commands Understood ] - - STATUS (none) -> - IDLE / BUILDING / WAITING / ABORTED - if BUILDING - BUILD-ID in progress - LOGTAIL (last 1kB or so of the log) - if WAITING - OK / DEPFAIL / PACKAGEFAIL / CHROOTFAIL / BUILDERFAIL - BUILD-ID waiting - List of filename <-> SHA1sum waiting for upload - if OK / PACKAGEFAIL - if ABORTED - BUILD-ID Aborted - - FETCHLOGTAIL (amount) -> - tail of log file (all of log file if amount <= 0) else (up-to) amount bytes - Returns an empty string if no build is in progress - - DOYOUHAVE (SHA1sum) -> - YES / NO - (May fetch from the librarian and return YES if it lacks the file) - - STOREFILE (CONTENT) -> - Stores the CONTENT and returns SHA1sum - - FETCHFILE (SHA1sum) -> - CONTENT for that SHA1summed file - - STARTBUILD (BUILD-ID, SHA1sum <-> filename list, CHROOT SHA1sum, - BUILDCOMMAND) -> - UNKNOWN-SUM - SHA1sum - UNKNOWN-BUILDER - INITIATED - - ABORT (none) -> - Any build in progress is aborted and the builder is placed in - ABORTED, returns ABORTED - - CLEAN (none) -> - The file-cache gets cleaned up, state is reverted to IDLE - returns IDLE - -An example interaction -^^^^^^^^^^^^^^^^^^^^^^ - -STATUS() -> IDLE -DOYOUHAVE(chrootsum) -> YES -DOYOUHAVE(dscsum) -> NO -STOREFILE(foo.dsc) -> dscsum -DOYOUHAVE(origsum) -> YES -DOYOUHAVE(diffsum) -> NO -STOREFILE(foo.diff.gz) -> diffsum -STARTBUILD(10023, { foo.dsc: dscsum, - foo.orig.tar: origsum, - foo.diff.gz: diffsum }, - chrootsum, DEBIAN-SOURCE) -> INITIATED -STATUS() -> BUILDING, 10023, "...logtail" - -...time passes... - -STATUS() -> BUILDING, 10023, "...logtail" - -...time passes... - -STATUS() -> WAITING, OK, 10023, foo.deb=debsum, foo.changes=changessum, - BUILDLOG=logsum -FETCHFILE(debsum) -> foo.deb -FETCHFILE(changessum) -> foo.changes -FETCHFILE(logsum) -> BUILDLOG -CLEAN() -> OK -STATUS() -> IDLE === removed file 'lib/canonical/buildd/sbuild-notes.txt' --- lib/canonical/buildd/sbuild-notes.txt 2009-12-16 00:16:24 +0000 +++ lib/canonical/buildd/sbuild-notes.txt 1970-01-01 00:00:00 +0000 @@ -1,49 +0,0 @@ -Notes on how to handle sbuild -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The buildd slave runs as the user buildd (ensure UID=1003, due 'tar' -silly defect when decompressing chroots) - -This user has sudo privs for running the chroots (must be passwordless) - -Sbuild needs patching: - -$main::build_source = 0; -+#$main::jobs_file = "$main::HOME/build/build-progress"; --$main::jobs_file = "build-progress"; -$main::max_lock_trys = 120; - - cleanup_close: - analyze_fail_stage( $pkgv ); -+ if( $main::pkg_status eq 'failed' ) { -+ $main::pkg_status = 'failed ' . $main::pkg_fail_stage; -+ } - write_jobs_file( $main::pkg_status ); - - close_log(); --unlink( $main::jobs_file ) if $main::batchmode; -+#unlink( $main::jobs_file ) if $main::batchmode; - unlink( "SBUILD-FINISHED" ) if $main::batchmode; - - -Process is as follows: - -1. unpack the chroot tarball (must unpack to chroot-autobuild) - Failure -> BUILDERFAIL -2. mount up the proc, and devpts dirs - Failure -> BUILDERFAIL -3. do apt-get update, apt-get dist-upgrade in the chroot - Failure -> CHROOTFAIL -4. Copy into the dir the relevant dsc, orig (and diff if needed) -5. sbuild --nolog --batch --distribution autobuild foo.dsc - Failure -> check build-progress - DEPFAIL / PACKAGEFAIL -6. sudo ./kill-any-chrooted-processes.py - Failure -> BUILDERFAIL -7. umount the proc, and devpts dirs - Failure -> BUILDERFAIL -8. unlink the chroot (rm -rf or whatever as root) - Failure -> BUILDERFAIL -9. OK - -We may make 8 an out-of-band process in which case it can't return BUILDERFAIL === removed file 'lib/canonical/buildd/uinotes.txt' --- lib/canonical/buildd/uinotes.txt 2007-10-31 19:03:15 +0000 +++ lib/canonical/buildd/uinotes.txt 1970-01-01 00:00:00 +0000 @@ -1,68 +0,0 @@ -Launchpad Build UI notes -^^^^^^^^^^^^^^^^^^^^^^^^ - -There are various parts to the buildd network which will have status -information which can be presented to various of the users. - -The following classes of users exist... - - o Boggy standard user - o Distribution build manager (member of some team or something) - o Package maintainer (independant) - o Package maintainer (in a distro context) - o Distribution security team member (can see hidden builds) - o Canonical buildd maintainer - -The buildd maintainer view -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The buildd maintainer wants to know if there are any builds failing -due to buildd-related issues. I.E. if any builds are returning -CHROOTFAIL or BUILDERFAIL - -The buildd maintainer also wants to see the build queues and the -states of each buildd in the farm. - -The boggy-standard user view -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -A plain and simple user (E.g. unauthenticated web access) can only see -build logs of unrestricted packages. They can see if a package is -queued to be built but no information on when it may be built. - -The independant package maintainer -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -An independant package maintainer will basically see the above and -also an ETA for each architecture's package build for their packages. - -For failed builds they can obviously see build logs. - -The package maintainer within a distro -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -A distro package maintainer gets to see the ETA of their package in -each of the arch queues; also what packages are ahead of it. - -Distribution security team member -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Gets to see all the above; but also for restricted distroseries - -Distribution build manager -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Gets to see an overview of all the queues and can give back packages -which managed to enter a manual dep-wait state or similar. - - -Pages we're likely to need -^^^^^^^^^^^^^^^^^^^^^^^^^^ - - o A view for the buildd maintainer. How to designate this I am - unsure; perhaps this ought to be a program to run to generate the - info rather than a web page. [[TODO: Ask lamont about this]] - o Build log viewer - o Build queue viewer, optionally limited to a specific - build/sourcepackagerelease/sourcepackage/distroseries/arch - o A way to give-back packages to re-try the build
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-reviewers Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-reviewers More help : https://help.launchpad.net/ListHelp
