It does work but statprof seems to stop getting samples. Last few lines from it:
2016-08-12 10:19:43+0000 [-] 0.00 0.06 0.00 banana.py:167:gotItem 2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00 policies.py:718:setTimeout 2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00 encoder.py:434:_iterencode 2016-08-12 10:19:43+0000 [-] --- 2016-08-12 10:19:43+0000 [-] Sample count: 51 2016-08-12 10:19:43+0000 [-] Total time: 0.510000 seconds The server is running. On Thu, Aug 11, 2016 at 10:46 AM, Francesco Di Mizio < [email protected]> wrote: > Pierre many thanks for that. I do not remember why the force scheduler api > didnt suite my needs, that was my first approach. > I will try the new snippet when I get to work in a few hours. > > On Wed, Aug 10, 2016 at 11:18 PM, Pierre Tardy <[email protected]> wrote: > >> Hi, >> >> After carefully looking at your code, it does not look to have obvious >> performance flaw. >> I think it would have been cheaper for you to just use the forcescheduler >> REST api, but if that code works for you this is totally fine. >> >> I saw that you did not manage to get sample in your prod as well, so I >> did look at my code again, and now I am able to see the samples. >> >> statprof does not like to stop before start >> >> Please try again with this code: >> >> >> import statprof >> from twisted.application.internet import TimerService >> from buildbot.util.service import BuildbotService >> class BuildbotTimerService(TimerService, BuildbotService): >> name = "timer" >> def __init__(self): >> BuildbotService.__init__(self) >> TimerService.__init__(self, 10, self.dump_stats) >> self.started = False >> def dump_stats(self): >> if not self.started: >> statprof.start() >> self.started = True >> return >> statprof.stop() >> with open("profile.txt", "w") as f: >> statprof.display(f) >> statprof.display() >> statprof.start() >> >> c['services'] = [ BuildbotTimerService() ] >> >> >> >> Le mer. 10 août 2016 à 14:14, Francesco Di Mizio < >> [email protected]> a écrit : >> >>> Many thanks. Find it attached. >>> >>> It's a tiny http server. People use a web page to send an HTTP request >>> to it to kick off builds. >>> >>> >>> On Wed, Aug 10, 2016 at 1:55 PM, Pierre Tardy <[email protected]> wrote: >>> >>>> I'll take a look. >>>> >>>> Le mer. 10 août 2016 12:45, Francesco Di Mizio < >>>> [email protected]> a écrit : >>>> >>>>> I am starting to think it may be due to my own custom scheduler. If I >>>>> uploaded the code somewhere could anybody take a look? it's a 150 lines py >>>>> script. >>>>> >>>>> On Wed, Aug 10, 2016 at 12:10 PM, Francesco Di Mizio < >>>>> [email protected]> wrote: >>>>> >>>>>> Thanks for that Pierre. I have it on on production and the >>>>>> performances do not degrade indeed. >>>>>> >>>>>> buildbot@e2123a50d1fb:~$ cat profile.txt >>>>>> No samples recorded. >>>>>> >>>>>> I need to study statprof a bit now ;) >>>>>> >>>>>> On Wed, Aug 10, 2016 at 11:20 AM, Vasily <[email protected]> wrote: >>>>>> >>>>>>> One can also use Intel® VTune™, you know... :-) >>>>>>> >>>>>>> Thanks, >>>>>>> Vasily >>>>>>> 09 авг. 2016 г. 23:56 пользователь "Pierre Tardy" <[email protected]> >>>>>>> написал: >>>>>>> >>>>>>>> You can add following snippet to your master.cfg. It looks statprof >>>>>>>> does not work on osx, so I cannot test it to the end, but I know >>>>>>>> similar >>>>>>>> did work on my prod >>>>>>>> >>>>>>>> >>>>>>>> import statprof >>>>>>>> from twisted.application.internet import TimerService >>>>>>>> from buildbot.util.service import BuildbotService >>>>>>>> class BuildbotTimerService(TimerService, BuildbotService): >>>>>>>> name = "timer" >>>>>>>> def __init__(self): >>>>>>>> BuildbotService.__init__(self) >>>>>>>> TimerService.__init__(self, 10, self.dump_stats) >>>>>>>> def dump_stats(self): >>>>>>>> statprof.stop() >>>>>>>> with open("profile.txt", "w") as f: >>>>>>>> statprof.display(f) >>>>>>>> statprof.start() >>>>>>>> >>>>>>>> c['services'] = [ BuildbotTimerService() ] >>>>>>>> >>>>>>>> >>>>>>>> Le mar. 9 août 2016 à 21:44, Francesco Di Mizio < >>>>>>>> [email protected]> a écrit : >>>>>>>> >>>>>>>>> Pierre, >>>>>>>>> >>>>>>>>> if you can enlighten on how you intend to use statprof, I will be >>>>>>>>> happy to give it a try. >>>>>>>>> This problem here is on top of my list right now as it's making my >>>>>>>>> prod env unusable. >>>>>>>>> >>>>>>>>> On Tue, Aug 9, 2016 at 8:25 PM, Pierre Tardy <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Please note that cprofile is very intrusive and will lead to >>>>>>>>>> about 3x slower code. >>>>>>>>>> >>>>>>>>>> So I would not run that in prod. This is why I prefer statprof >>>>>>>>>> which is using a non intrusive method to get the stats >>>>>>>>>> >>>>>>>>>> Le mar. 9 août 2016 19:15, Francesco Di Mizio < >>>>>>>>>> [email protected]> a écrit : >>>>>>>>>> >>>>>>>>>>> On a side note I have managed to get twistd to dump a blob I can >>>>>>>>>>> then load and analyze. Got to see if I can run this in production. >>>>>>>>>>> If anybody is curious the following worked for me: >>>>>>>>>>> twistd --savestats -n --profiler=cprofile --profile=/crcdata/ >>>>>>>>>>> profile.stats -y ./buildbot.tac >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 9, 2016 at 5:21 PM, Francesco Di Mizio < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Well that's expected to an extent - p4 poller is running 'p4 >>>>>>>>>>>> changes' every pollinterval seconds. >>>>>>>>>>>> >>>>>>>>>>>> Anyway just as a test I've tried to disable the poller and >>>>>>>>>>>> still at times I am seeing the CPU spiking up to above 100%. I >>>>>>>>>>>> believe I >>>>>>>>>>>> really need to profile this somehow. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 9, 2016 at 5:08 PM, Dan Kegel <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Also watch 'top' and see if poller processes are hogging >>>>>>>>>>>>> resources. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> [email protected] >>>>>>>> https://lists.buildbot.net/mailman/listinfo/users >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >
_______________________________________________ users mailing list [email protected] https://lists.buildbot.net/mailman/listinfo/users
