On Fri, 2021-10-29 at 09:22 -0400, Trevor Gamblin wrote: > We need a way to limit the builds for when a given worker has less than > a certain amount of disk space available. This implements a > canStartBuild method based on the example in the Buildbot docs and > blocks a build if the worker has less than 60GB of disk space available. > Unlike the example code, we want the stdout of the command so that we > can calculate the amount of disk space, rather than just relying on the > remote command's return code. > > Docs: > https://docs.buildbot.net/latest/manual/customization.html#canstartbuild-functions > > [YOCTO #14591] > > Signed-off-by: Trevor Gamblin <[email protected]> > --- > builders.py | 44 ++++++++++++++++++++++++++++++++++++++++---- > 1 file changed, 40 insertions(+), 4 deletions(-) > > diff --git a/builders.py b/builders.py > index 5773950..0d1facc 100644 > --- a/builders.py > +++ b/builders.py > @@ -8,6 +8,7 @@ from yoctoabb import config > from yoctoabb.steps.writelayerinfo import WriteLayerInfo > from yoctoabb.steps.runconfig import get_publish_dest, > get_publish_resultdir, get_publish_name, RunConfigCheckSteps, TargetPresent > from buildbot.process.results import Results, SUCCESS, FAILURE, CANCELLED, > WARNINGS, SKIPPED, EXCEPTION, RETRY > +from buildbot.process.remotecommand import RemoteCommand > > from twisted.python import log > from twisted.internet import defer > @@ -45,6 +46,41 @@ def ensure_props_set(props): > "publish_destination": props.getProperty("publish_destination", "") > } > > [email protected] > +def shell(command, worker, builder): > + args = { > + 'command': command, > + 'workdir': worker.worker_basedir, > + 'logEnviron': False, > + 'want_stdout': True, > + 'want_stderr': False, > + } > + > + cmd = RemoteCommand('shell', args, collectStdout=True, > stdioLogName="stdio") > + cmd.worker = worker > + yield cmd.run(None, worker.conn, builder.name) > + return cmd > + > [email protected] > +def canStartBuild(builder, wfb, request): > + log.msg("Checking available disk space...") > + > + cmd = yield shell("df -BG | grep $(findmnt -T . | awk '{print $2}' | sed > -n 2p) | awk '{print $4}' | sed 's/[^0-9]*//g'", wfb.worker, builder) > + threshold = 60 # GB of space > + if int(cmd.stdout) < threshold: > + log.msg("Detected {0} GB of space available, less than threshold of > {1} GB. Can't start build".format(cmd.stdout, threshold)) > + wfb.worker.putInQuarantine() > + return False > + else: > + log.msg("Detected {0} GB of space available, more than threshold of > {1} GB. OK to build".format(cmd.stdout, threshold)) > + > + wfb.worker.quarantine_timeout = 120 > + wfb.worker.putInQuarantine() > + > + wfb.worker.resetQuarantine() > + > + return True > +
Unfortunately this quarantine piece is causing problems. It means that even if there is free space on the builder and always was, a maximum of one build every 2 minutes can be started. I'll probably drop the quarantine piece as it was a nice to have soft start for recovery rather than an essential. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#55192): https://lists.yoctoproject.org/g/yocto/message/55192 Mute This Topic: https://lists.yoctoproject.org/mt/86676252/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
