Hy Francesco,
This is weird, as on my environment I can see the sample count growing. It
only grows if the process is not idle, though
So I have to use the master ui to actually make the profile change on my
test environment.
Your profile looks like fairly idle. I can see pprint and write to logfile
which might be not scalable if the data start growing do you have some
heavy debug log in your prod?
Here is an updated more simple snippet, but this should not change many
thing.
You can improve the TimerService '10' value to something bigger in order to
dump less often.
Are you sure you did reproduce the slowness while profiling?
import statprof
from twisted.application.internet import TimerService
from buildbot.util.service import BuildbotService
from twisted.internet import reactor
class BuildbotTimerService(TimerService, BuildbotService):
name = "timer"
def __init__(self):
BuildbotService.__init__(self)
TimerService.__init__(self, 10, self.dump_stats)
reactor.callFromThread(statprof.start)
def dump_stats(self):
statprof.stop()
with open("profile.txt", "w") as f:
statprof.display(f)
statprof.display()
statprof.start()
c['services'] = [ BuildbotTimerService() ]
Le ven. 12 août 2016 à 14:21, Francesco Di Mizio <[email protected]>
a écrit :
> Sure thing. it's attached.
> Please note the sampe count has been stuck at 51 for over 1 hour even if
> the master is running.
>
> On Fri, Aug 12, 2016 at 12:38 PM, Pierre Tardy <[email protected]> wrote:
>
>> The first lines are the important ones. Can you send the full file?
>>
>> Le ven. 12 août 2016 12:23, Francesco Di Mizio <
>> [email protected]> a écrit :
>>
>>> It does work but statprof seems to stop getting samples. Last few lines
>>> from it:
>>>
>>> 2016-08-12 10:19:43+0000 [-] 0.00 0.06 0.00
>>> banana.py:167:gotItem
>>> 2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00
>>> policies.py:718:setTimeout
>>> 2016-08-12 10:19:43+0000 [-] 0.00 0.01 0.00
>>> encoder.py:434:_iterencode
>>> 2016-08-12 10:19:43+0000 [-] ---
>>> 2016-08-12 10:19:43+0000 [-] Sample count: 51
>>> 2016-08-12 10:19:43+0000 [-] Total time: 0.510000 seconds
>>>
>>> The server is running.
>>>
>>> On Thu, Aug 11, 2016 at 10:46 AM, Francesco Di Mizio <
>>> [email protected]> wrote:
>>>
>>>> Pierre many thanks for that. I do not remember why the force scheduler
>>>> api didnt suite my needs, that was my first approach.
>>>> I will try the new snippet when I get to work in a few hours.
>>>>
>>>> On Wed, Aug 10, 2016 at 11:18 PM, Pierre Tardy <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> After carefully looking at your code, it does not look to have obvious
>>>>> performance flaw.
>>>>> I think it would have been cheaper for you to just use the
>>>>> forcescheduler REST api, but if that code works for you this is totally
>>>>> fine.
>>>>>
>>>>> I saw that you did not manage to get sample in your prod as well, so I
>>>>> did look at my code again, and now I am able to see the samples.
>>>>>
>>>>> statprof does not like to stop before start
>>>>>
>>>>> Please try again with this code:
>>>>>
>>>>>
>>>>> import statprof
>>>>> from twisted.application.internet import TimerService
>>>>> from buildbot.util.service import BuildbotService
>>>>> class BuildbotTimerService(TimerService, BuildbotService):
>>>>> name = "timer"
>>>>> def __init__(self):
>>>>> BuildbotService.__init__(self)
>>>>> TimerService.__init__(self, 10, self.dump_stats)
>>>>> self.started = False
>>>>> def dump_stats(self):
>>>>> if not self.started:
>>>>> statprof.start()
>>>>> self.started = True
>>>>> return
>>>>> statprof.stop()
>>>>> with open("profile.txt", "w") as f:
>>>>> statprof.display(f)
>>>>> statprof.display()
>>>>> statprof.start()
>>>>>
>>>>> c['services'] = [ BuildbotTimerService() ]
>>>>>
>>>>>
>>>>>
>>>>> Le mer. 10 août 2016 à 14:14, Francesco Di Mizio <
>>>>> [email protected]> a écrit :
>>>>>
>>>>>> Many thanks. Find it attached.
>>>>>>
>>>>>> It's a tiny http server. People use a web page to send an HTTP
>>>>>> request to it to kick off builds.
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 10, 2016 at 1:55 PM, Pierre Tardy <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> I'll take a look.
>>>>>>>
>>>>>>> Le mer. 10 août 2016 12:45, Francesco Di Mizio <
>>>>>>> [email protected]> a écrit :
>>>>>>>
>>>>>>>> I am starting to think it may be due to my own custom scheduler. If
>>>>>>>> I uploaded the code somewhere could anybody take a look? it's a 150
>>>>>>>> lines
>>>>>>>> py script.
>>>>>>>>
>>>>>>>> On Wed, Aug 10, 2016 at 12:10 PM, Francesco Di Mizio <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Thanks for that Pierre. I have it on on production and the
>>>>>>>>> performances do not degrade indeed.
>>>>>>>>>
>>>>>>>>> buildbot@e2123a50d1fb:~$ cat profile.txt
>>>>>>>>> No samples recorded.
>>>>>>>>>
>>>>>>>>> I need to study statprof a bit now ;)
>>>>>>>>>
>>>>>>>>> On Wed, Aug 10, 2016 at 11:20 AM, Vasily <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> One can also use Intel® VTune™, you know... :-)
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Vasily
>>>>>>>>>> 09 авг. 2016 г. 23:56 пользователь "Pierre Tardy" <
>>>>>>>>>> [email protected]> написал:
>>>>>>>>>>
>>>>>>>>>>> You can add following snippet to your master.cfg. It looks
>>>>>>>>>>> statprof does not work on osx, so I cannot test it to the end, but
>>>>>>>>>>> I know
>>>>>>>>>>> similar did work on my prod
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> import statprof
>>>>>>>>>>> from twisted.application.internet import TimerService
>>>>>>>>>>> from buildbot.util.service import BuildbotService
>>>>>>>>>>> class BuildbotTimerService(TimerService, BuildbotService):
>>>>>>>>>>> name = "timer"
>>>>>>>>>>> def __init__(self):
>>>>>>>>>>> BuildbotService.__init__(self)
>>>>>>>>>>> TimerService.__init__(self, 10, self.dump_stats)
>>>>>>>>>>> def dump_stats(self):
>>>>>>>>>>> statprof.stop()
>>>>>>>>>>> with open("profile.txt", "w") as f:
>>>>>>>>>>> statprof.display(f)
>>>>>>>>>>> statprof.start()
>>>>>>>>>>>
>>>>>>>>>>> c['services'] = [ BuildbotTimerService() ]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Le mar. 9 août 2016 à 21:44, Francesco Di Mizio <
>>>>>>>>>>> [email protected]> a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Pierre,
>>>>>>>>>>>>
>>>>>>>>>>>> if you can enlighten on how you intend to use statprof, I will
>>>>>>>>>>>> be happy to give it a try.
>>>>>>>>>>>> This problem here is on top of my list right now as it's making
>>>>>>>>>>>> my prod env unusable.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 9, 2016 at 8:25 PM, Pierre Tardy <[email protected]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Please note that cprofile is very intrusive and will lead to
>>>>>>>>>>>>> about 3x slower code.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I would not run that in prod. This is why I prefer statprof
>>>>>>>>>>>>> which is using a non intrusive method to get the stats
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le mar. 9 août 2016 19:15, Francesco Di Mizio <
>>>>>>>>>>>>> [email protected]> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On a side note I have managed to get twistd to dump a blob I
>>>>>>>>>>>>>> can then load and analyze. Got to see if I can run this in
>>>>>>>>>>>>>> production.
>>>>>>>>>>>>>> If anybody is curious the following worked for me:
>>>>>>>>>>>>>> twistd --savestats -n --profiler=cprofile --profile=/crcdata/
>>>>>>>>>>>>>> profile.stats -y ./buildbot.tac
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 9, 2016 at 5:21 PM, Francesco Di Mizio <
>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Well that's expected to an extent - p4 poller is running 'p4
>>>>>>>>>>>>>>> changes' every pollinterval seconds.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyway just as a test I've tried to disable the poller and
>>>>>>>>>>>>>>> still at times I am seeing the CPU spiking up to above 100%. I
>>>>>>>>>>>>>>> believe I
>>>>>>>>>>>>>>> really need to profile this somehow.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 9, 2016 at 5:08 PM, Dan Kegel <[email protected]>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also watch 'top' and see if poller processes are hogging
>>>>>>>>>>>>>>>> resources.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> [email protected]
>>>>>>>>>>> https://lists.buildbot.net/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>
>
_______________________________________________
users mailing list
[email protected]
https://lists.buildbot.net/mailman/listinfo/users
