The ram looks like sufficient, it might be a good test to try and increase the number of cpu for that VM. In your trace, I can see the use of up to 7 threads at the same time, so you might gain by going 8 CPUs
Also, make sure that your VM host is not overbooked. In my experience of using VMware VMs provided by IT, overbooking has been a source of inexplicable performance issues. 12k lines is a lot, but buildbot shall support this kind of load without issue. Le ven. 26 août 2016 à 12:05, Francesco Di Mizio <[email protected]> a écrit : > It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs@3Ghz. It runs, > among other marginal things, 2 docker containers - one for the buildbot and > one for the postgres db. > > The most beefy logs have around 12K lines. Is it too much? > Also some other logs are read from the worker's filesystem and added as > additional logs. > > On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <[email protected]> wrote: > >> Cool! >> I can indeed see 3 spikes. >> >> Looks related to logs and logs compression. >> >> What is the HW spec of your master machine? >> How much log does your build generate? >> >> Pierre >> >> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio < >> [email protected]> a écrit : >> >>> Pierre, >>> >>> I enabled it, waited 1 min and saw the spike, then stopped after a few >>> secs. Attached the json. >>> Awesome tool btw, work wonders! >>> >>> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <[email protected]> wrote: >>> >>>> You can try to hit the button in the morning with a two hours gather >>>> period, and hope that you see the spike during that period.. >>>> >>>> >>>> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio < >>>> [email protected]> a écrit : >>>> >>>>> Thanks a lot! Pierre I will def will give it a shot. I am not sure >>>>> I'll be able to smash that 'start recording' button as the UI >>>>> isusuallystuck when the CPU spikes. Updates to come! >>>>> >>>>> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Francesco, >>>>>> >>>>>> I spent some time in order to implement a profiler plugin for buildbot >>>>>> >>>>>> You can give it a look, and send your profile.json file if you need >>>>>> more analysis from me. >>>>>> https://github.com/tardyp/buildbot_profiler >>>>>> >>>>>> Regards, >>>>>> Pierre >>>>>> >>>>>> >>>>>> Le mer. 24 août 2016 à 22:43, Francesco Di Mizio < >>>>>> [email protected]> a écrit : >>>>>> >>>>>>> I've tried and it's not an easy task because of my Win into Vagrant >>>>>>> into Docker setup. >>>>>>> I'll try again soon when I get a Linux box! >>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 5:54 PM, Vasily <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Umm, no. VTune has Python support starting 2017 Beta, and, well, it >>>>>>>> was my team (at Intel) work actually :-) >>>>>>>> >>>>>>>> P.S. I'm from Intel, too. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Vasily >>>>>>>> 19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" < >>>>>>>> [email protected]> написал: >>>>>>>> >>>>>>>> I had thought you were making fun of Intel somehow ;) >>>>>>>>> >>>>>>>>> On Aug 19, 2016 5:07 PM, "Pierre Tardy" <[email protected]> wrote: >>>>>>>>> >>>>>>>>> ahah >>>>>>>>> >>>>>>>>> I though this was a taunt on me being employed by Intel. >>>>>>>>> I actually had mitigated experience with vtune few years ago, and >>>>>>>>> didn't know they had python support until then. >>>>>>>>> Being an opensource guy, I usually neglegate to look at >>>>>>>>> proprietary stuff. >>>>>>>>> >>>>>>>>> Pierre >>>>>>>>> >>>>>>>>> Le ven. 19 août 2016 à 12:18, Vasily <[email protected]> a >>>>>>>>> écrit : >>>>>>>>> >>>>>>>>>> I'm again suggesting to look into Python profiling capabilities >>>>>>>>>> of Intel® VTune™ Amplifier. It could run statistical profiling for a >>>>>>>>>> long >>>>>>>>>> time and display CPU usage over time, so the developer can look at >>>>>>>>>> specific >>>>>>>>>> time range where CPU usage was too high and see which functions were >>>>>>>>>> executed. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Vasily >>>>>>>>>> 19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" < >>>>>>>>>> [email protected]> написал: >>>>>>>>>> >>>>>>>>>> Hi Francesco, >>>>>>>>>>> >>>>>>>>>>> Your described setup looks sane to me. >>>>>>>>>>> >>>>>>>>>>> The problems we are trying to catch are cpu spikes, as far as I >>>>>>>>>>> understand, which does not happen for very long, but are very >>>>>>>>>>> annoying for >>>>>>>>>>> users, as it is blocking the reactor. >>>>>>>>>>> >>>>>>>>>>> This problem is not easy to see in the profile you sent, as this >>>>>>>>>>> profile is over long time, so we see the average of each method >>>>>>>>>>> during the >>>>>>>>>>> day and not the spikes. >>>>>>>>>>> >>>>>>>>>>> What would really be needed is a on-demand profiler which would >>>>>>>>>>> detect cpu spikes and only log the stack traces during those times. >>>>>>>>>>> >>>>>>>>>>> Here is a nice blog pst explaining why statistic profiling is >>>>>>>>>>> cool and easy to implement in python. >>>>>>>>>>> https://nylas.com/blog/performance >>>>>>>>>>> >>>>>>>>>>> For 0.9.1 I want to concentrate on scalability, and write a >>>>>>>>>>> debugging ui plugin based on those ideas (and probably code) >>>>>>>>>>> >>>>>>>>>>> That would be great if your team can help on that matter. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Pierre >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >
_______________________________________________ users mailing list [email protected] https://lists.buildbot.net/mailman/listinfo/users
