Hello,
A lot of things happen between 2.7 and 2.10, although I don't see
anything
which could impact the performance that much. (maybe new reporter
framework, but really not convinced)
If you see that the db is underutilized this must be a classical reactor
starvation.
With asynchronous systems like buildbot, you shouldn't do any heavy
computation in the main event loop thread, those must be done in a
thread
via deferToThread and co.
Those are the common issues you can have with performance
independantly from upgrade regressions:
1) Custom steps:
A lot of time, we see people struggling with performance when they just
have some custom step doing heavy computation that block the main thread
constantly, preventing all the very quick tasks to run in //.
2) too much logs
In this case, there is not much to do beside reducing the log amount.
This
would be the time to switch to a multi-master setup, where you put 2
masters for builds, and one master for web UI.
You can put those in the same machine/VM, no problem, the only work is
to
have separate processes that each have several event queues. You can use
docker-compose or kubernetes in order to more easily create such
deployment. We don't have readily useable for that, but several people
have
done and documented it, for example
https://github.com/pop/buildbot-on-kubernetes
I have developed the buildbot profiler in order to quickly find those.
You
just have to install it as a plugin and start a profile whenever the
buildbot feels slow.
It is a statistical profiler, so it will not significantly change the
actual performance so it is safe to run in production.
https://pypi.org/project/buildbot-profiler/
Regards,
Pierre
Le mar. 12 janv. 2021 à 01:29, Yngve N. Pettersen <[email protected]> a
écrit :
Hello all,
We have just upgraded our buildbot system from 2.7 to 2.10.
However, I am noticing performance issues when loading these pages:
Builds->Builders
Builds->Workers
individual builds
Loading these can take several minutes, although there are periods of
immediate responses.
What I am seeing on the buildbot manager machine is that the Python3
process hits 90-100% for the entire period.
The Python version is 3.6.9 running on Ubuntu 18.04
As far as I can tell, the Postgresql database is mostly idle during
this
period. I did do a full vacuum a few hours ago, in case that was the
issue.
There are about 40 builders, and 30 workers in the system, only about
10-15 of these have a 10-20 builds for the past few days, although most
of
these have active histories of 3000 builds (which do make me wonder if
the
problem could be a lack of limiting the DB queries, at present I have
not
inspected the DB queries).
The individual builds can have very large log files in the build steps,
in
many cases tens of thousands of lines (we _are_ talking about a
Chromium
based project).
Our changes in the builders and workers JS code are minimal (we are
using
a custom build of www-base), just using different information for the
build labels (build version number), and grouping the builders, which
should not be causing any performance issues. (we have larger changes
in
the individual builder view, where we include Git commit messages, and
I
have so far not seen any performance issues there)
BTW: The line plots for build time and successes on builders seems to
be
MIA. Not sure if that is an upstream issue, or due to something in our
www-base build.
Do you have any suggestions for where to look for the cause of the
problem?
--
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS
_______________________________________________
users mailing list
[email protected]
https://lists.buildbot.net/mailman/listinfo/users
