I just went ahead and disabled the potentially too long logs and havent seen a spice since then ;) finger crossed!
On Fri, Aug 26, 2016 at 5:35 PM, Pierre Tardy <[email protected]> wrote: > Hum, > It might not be stuck, actually, but just spending very long time to > compress the log. > > In theory, the log compression is not waited for, though. > > Note that if you stop a build that is waiting for a lost deferred, this > will have no effect as you describe. > > You got absolutly no exception in twisted.log? > > Pierre > > > Le ven. 26 août 2016 à 17:06, Neil Gilmore <[email protected]> a > écrit : > >> I'm currently looking at a step with 3 logs: >> 81531 lines >> 489285 lines >> 489311 lines >> >> An earlier successful run would have that first log at 244080 lines. >> >> This particular build is stuck, though. :( (which is why I'm looking at >> it.) (A bit off-topic, but I tried stopping it. The last step is marked >> cancelled, but that's the only effect.) >> >> >> Neil Gilmore >> grammatech.com >> >> >> On 8/26/2016 6:25 AM, Pierre Tardy wrote: >> >> >> The ram looks like sufficient, it might be a good test to try and >> increase the number of cpu for that VM. >> In your trace, I can see the use of up to 7 threads at the same time, so >> you might gain by going 8 CPUs >> >> Also, make sure that your VM host is not overbooked. In my experience of >> using VMware VMs provided by IT, overbooking has been a source of >> inexplicable performance issues. >> >> 12k lines is a lot, but buildbot shall support this kind of load without >> issue. >> >> Le ven. 26 août 2016 à 12:05, Francesco Di Mizio < >> [email protected]> a écrit : >> >>> It's a vmware virtual machine with 4 GIGs RAM and 4 CPUs@3Ghz. It runs, >>> among other marginal things, 2 docker containers - one for the buildbot and >>> one for the postgres db. >>> >>> The most beefy logs have around 12K lines. Is it too much? >>> Also some other logs are read from the worker's filesystem and added as >>> additional logs. >>> >>> On Fri, Aug 26, 2016 at 11:51 AM, Pierre Tardy <[email protected]> wrote: >>> >>>> Cool! >>>> I can indeed see 3 spikes. >>>> >>>> Looks related to logs and logs compression. >>>> >>>> What is the HW spec of your master machine? >>>> How much log does your build generate? >>>> >>>> Pierre >>>> >>>> Le ven. 26 août 2016 à 11:42, Francesco Di Mizio < >>>> [email protected]> a écrit : >>>> >>>>> Pierre, >>>>> >>>>> I enabled it, waited 1 min and saw the spike, then stopped after a few >>>>> secs. Attached the json. >>>>> Awesome tool btw, work wonders! >>>>> >>>>> On Thu, Aug 25, 2016 at 1:14 PM, Pierre Tardy <[email protected]> >>>>> wrote: >>>>> >>>>>> You can try to hit the button in the morning with a two hours gather >>>>>> period, and hope that you see the spike during that period.. >>>>>> >>>>>> >>>>>> Le jeu. 25 août 2016 à 12:17, Francesco Di Mizio < >>>>>> [email protected]> a écrit : >>>>>> >>>>>>> Thanks a lot! Pierre I will def will give it a shot. I am not sure >>>>>>> I'll be able to smash that 'start recording' button as the UI >>>>>>> isusuallystuck when the CPU spikes. Updates to come! >>>>>>> >>>>>>> On Thu, Aug 25, 2016 at 10:45 AM, Pierre Tardy <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Francesco, >>>>>>>> >>>>>>>> I spent some time in order to implement a profiler plugin for >>>>>>>> buildbot >>>>>>>> >>>>>>>> You can give it a look, and send your profile.json file if you need >>>>>>>> more analysis from me. >>>>>>>> https://github.com/tardyp/buildbot_profiler >>>>>>>> >>>>>>>> Regards, >>>>>>>> Pierre >>>>>>>> >>>>>>>> >>>>>>>> Le mer. 24 août 2016 à 22:43, Francesco Di Mizio < >>>>>>>> [email protected]> a écrit : >>>>>>>> >>>>>>>>> I've tried and it's not an easy task because of my Win into >>>>>>>>> Vagrant into Docker setup. >>>>>>>>> I'll try again soon when I get a Linux box! >>>>>>>>> >>>>>>>>> On Fri, Aug 19, 2016 at 5:54 PM, Vasily <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Umm, no. VTune has Python support starting 2017 Beta, and, well, >>>>>>>>>> it was my team (at Intel) work actually :-) >>>>>>>>>> >>>>>>>>>> P.S. I'm from Intel, too. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Vasily >>>>>>>>>> 19 авг. 2016 г. 18:17 пользователь "Francesco Di Mizio" < >>>>>>>>>> [email protected]> написал: >>>>>>>>>> >>>>>>>>>> I had thought you were making fun of Intel somehow ;) >>>>>>>>>>> >>>>>>>>>>> On Aug 19, 2016 5:07 PM, "Pierre Tardy" <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> ahah >>>>>>>>>>> >>>>>>>>>>> I though this was a taunt on me being employed by Intel. >>>>>>>>>>> I actually had mitigated experience with vtune few years ago, >>>>>>>>>>> and didn't know they had python support until then. >>>>>>>>>>> Being an opensource guy, I usually neglegate to look at >>>>>>>>>>> proprietary stuff. >>>>>>>>>>> >>>>>>>>>>> Pierre >>>>>>>>>>> >>>>>>>>>>> Le ven. 19 août 2016 à 12:18, Vasily <[email protected]> a >>>>>>>>>>> écrit : >>>>>>>>>>> >>>>>>>>>>>> I'm again suggesting to look into Python profiling capabilities >>>>>>>>>>>> of Intel® VTune™ Amplifier. It could run statistical profiling for >>>>>>>>>>>> a long >>>>>>>>>>>> time and display CPU usage over time, so the developer can look at >>>>>>>>>>>> specific >>>>>>>>>>>> time range where CPU usage was too high and see which functions >>>>>>>>>>>> were >>>>>>>>>>>> executed. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Vasily >>>>>>>>>>>> 19 авг. 2016 г. 11:57 пользователь "Pierre Tardy" < >>>>>>>>>>>> [email protected]> написал: >>>>>>>>>>>> >>>>>>>>>>>> Hi Francesco, >>>>>>>>>>>>> >>>>>>>>>>>>> Your described setup looks sane to me. >>>>>>>>>>>>> >>>>>>>>>>>>> The problems we are trying to catch are cpu spikes, as far as >>>>>>>>>>>>> I understand, which does not happen for very long, but are very >>>>>>>>>>>>> annoying >>>>>>>>>>>>> for users, as it is blocking the reactor. >>>>>>>>>>>>> >>>>>>>>>>>>> This problem is not easy to see in the profile you sent, as >>>>>>>>>>>>> this profile is over long time, so we see the average of each >>>>>>>>>>>>> method during >>>>>>>>>>>>> the day and not the spikes. >>>>>>>>>>>>> >>>>>>>>>>>>> What would really be needed is a on-demand profiler which >>>>>>>>>>>>> would detect cpu spikes and only log the stack traces during >>>>>>>>>>>>> those times. >>>>>>>>>>>>> >>>>>>>>>>>>> Here is a nice blog pst explaining why statistic profiling is >>>>>>>>>>>>> cool and easy to implement in python. >>>>>>>>>>>>> https://nylas.com/blog/performance >>>>>>>>>>>>> >>>>>>>>>>>>> For 0.9.1 I want to concentrate on scalability, and write a >>>>>>>>>>>>> debugging ui plugin based on those ideas (and probably code) >>>>>>>>>>>>> >>>>>>>>>>>>> That would be great if your team can help on that matter. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Pierre >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> >> >> _______________________________________________ >> users mailing >> [email protected]https://lists.buildbot.net/mailman/listinfo/users >> >> _______________________________________________ >> users mailing list >> [email protected] >> https://lists.buildbot.net/mailman/listinfo/users > > > _______________________________________________ > users mailing list > [email protected] > https://lists.buildbot.net/mailman/listinfo/users >
_______________________________________________ users mailing list [email protected] https://lists.buildbot.net/mailman/listinfo/users
