Hi, Posting a follow up: We've now established a multi master setup, with two masters: 1st handles the web interface and force schedulers 2nd handles everything else
It has fixed nearly all our performance issues in the web interface, and after a week we are not experiencing issues. The Changes API is still the slowest and can occasionally take 10-20 second to respond, but most other APIs have gone from multiple seconds to a few milliseconds. Thanks a lot for your help and continuous work on Buildbot. Best regards, Jan On 2020-09-21 21:31, Pierre Tardy wrote:
Hi Jan, Indeed multimaster will probably help for your scalability issues. For performance monitoring, what takes 5min? Normally the UI is divided into several api calls, so it might slowly build, or is there 5min wait before displaying anything? You might want to use https://pypi.org/project/buildbot-profiler/ in order to see what function in buildbot is using too much CPU. Also, what can impact performance is some blocking call made inside custom steps. For example, if you are using the `requests` library for doing http calls, then this could impact a lot of the performance. Those should be replaced by equivalent async (treq/txrequest) Regards Pierre Le ven. 18 sept. 2020 à 17:33, Jan Marker <[email protected] <mailto:[email protected]>> a écrit : Hello, I was wondering whether you have any experience/advise about a two master configuration, where one master is serving the web service and the other is coordinating builds as mentioned in the documentation [0]. We're talking of a setup with about 500-1500 builds of about 140 builders per day. The web interface can take up to a minute to respond, certain pages might take up to 5 minutes. The system is using a MySQL database and presumably reasonable caching values, but we do see multi-minute delays from buildrequest arrival to build start and the aforementioned web interface performance. So were wondering whether that kind of multi-master setup might help us out and improve the performance metrics. I'd appreciate any kind of advice or experience with multi-master setups. Thanks, Jan Marker [0] http://docs.buildbot.net/2.8.4/manual/concepts.html#multimaster-- Jan Marker | [email protected] <mailto:[email protected]> | Software EngineerKDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts _______________________________________________ users mailing list [email protected] <mailto:[email protected]> https://lists.buildbot.net/mailman/listinfo/users
-- Jan Marker | [email protected] | Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt, C++ and OpenGL Experts
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ users mailing list [email protected] https://lists.buildbot.net/mailman/listinfo/users
