On May 29, 2019, at 13:29, Mojca Miklavec wrote:
> As an intermezzo I want to explain one thing. Initially we had one single job
> (which could run for days), and it built all the ports in a single job.
> However one could not infer any information about what worked and what failed
> without downloading logs first, which could have reached hundreds of
> megabytes each. We wanted to rewrite it with a custom number of steps, so
> that a build of 100 ports would contain 100 steps (probably more like a few
> times hundred; with "clean", "build dependencies", "build port", "upload
> port", ... for each of the ports individually). The reason for creating port
> watcher and port builder as separate entities was used as a workaround for
> lack of functionality in buildbot 0.8, which is no longer the case in the
> latest buildbot. So maybe it's even time to revise this decision and figure
> out if there's a better way to organize port watcher/builder into a single
> builder.
Having a portwatcher and a portbuilder for each OS version causes many
problems. I'm all for combining them. We already discussed my proposal for
doing that:
https://lists.macports.org/pipermail/macports-dev/2018-March/037577.html
> One problem with current setup is that it may happen that one presses the
> "rebuild" button and starts building a port on the portbuilder at the same
> time as portindex is being regenerated, or macports is being updated in the
> portwatcher. These two actions should not happen at the same time.
This is not a problem currently because each portwatcher build waits for all of
its portbuilder builds to finish. It is only currently a problem when there is
a power outage or a buildworker crashes. It is annoying when that happens and I
have to monitor the builds after everything is back up and usually reschedule
some builds that failed because a portindex was being synced during a build,
but fortunately it doesn't happen often.
> Another problem is that, say, a perl version increase may result in
> rebuilding a few thousand ports. Buildbot 0.8 usually choked and became
> uselessly slow (with builders idling for as long as an hour just to get the
> next job assigned).
This was only a problem because our buildmaster's disk is slow and our buildbot
cache settings were too low. We increased the cache sizes so that the pending
builds can all be kept in RAM, so this isn't really a problem anymore, as far
as I know.
> Before that rewrite, when everything ran as a single job, we would sometimes
> schedule builds of all ports. With current setup it's more of a mission
> impossible to imagine that. I have no idea how buildbot 2.x can handle builds
> with tens of thousands of steps.
I touched on the problem of occasional huge numbers of builds in my combining
proposal above. Builds of huge numbers of ports (whether forced or via a mass
commit) could receive a "lesser priority" so that subsequent "normal" commits
of one or a small number of ports could receive the normal "higher priority"
and "skip the queue" and go first.
