Thanks for the kind replies both of you. @Pierre: Not sure I get what you mean. Given the context, for a step to be CPU demanding it should be a master side step right? I happen to not have any. What would you be profiling with statprof? I'd really appreciate if you could elaborate on your idea.
Really all I can think of is the poller. I'll keep looking into it. On Tue, Aug 2, 2016 at 6:36 PM, Dan Kegel <[email protected]> wrote: > With gitpoller, it was easy to see; whenever the number of git > sessions from the poller went over 0 or so, web gui performance was > poor. > And if it went over 10, well, you could kiss the gui goodbye for > several minutes. > > One countermeasure was to randomize the polling intervals, a la > > interval=6 # minutes > self['change_source'].append( > # Fuzz the interval to avoid slamming the git server > and hitting the MaxStartups or MaxSessions limits > # If you hit them, twistd.log will have lots of > "ssh_exchange_identification: Connection closed by remote host" errors > # See http://trac.buildbot.net/ticket/2480 > changes.GitPoller(repourl, branches=branchnames, > workdir='gitpoller-workdir-'+name, pollinterval=interval*60 + > random.uniform(-10, 10))) > > That made life just barely bearable, at least until number of projects > polled was under 50 or so. > What really helped was not using pollers anymore, and switching to > gitlab's webhooks. > We're at 190 now, of which 57 are still using gitpoller, and it's > almost ok. (I really have > to move the last 57 onto gitlab. Or, well, since they're not > critical, increase the polling > interval...) > > On Tue, Aug 2, 2016 at 9:13 AM, Pierre Tardy <[email protected]> wrote: > > Hi, > > > > Pollers are usually indeed not scaling as they, hmm, poll. > > What you are describing here is hints that the twisted reactor thread is > > always busy, which should not happen if you only start 10 builds. > > You might have some custom steps which are doing something heavily cpu > bound > > in the main thread. > > What I usually do is to use statprof: > > https://pypi.python.org/pypi/statprof/ > > > > in order to know what the cpu is doing. > > You could create a builder which you can trig whenever you need, and > which > > would start the profiling, wait a few minutes, and then save profiling > to a > > file. > > > > > > > > Le mar. 2 août 2016 à 17:53, Francesco Di Mizio < > [email protected]> > > a écrit : > >> > >> Hey Dan, > >> > >> I am using a p4 poller. Maybe it's suffering from the same problems? > >> > >> On Tue, Aug 2, 2016 at 5:45 PM, Francesco Di Mizio > >> <[email protected]> wrote: > >>> > >>> I'd like to provide a bit more context.Right after restarting the > master > >>> and kicking off 10 builds CPU was at 110-120%. This made the UI > unusable and > >>> basically all the services were stuck, including the REST API. > >>> After 3-4 minutes like this and WITH all the 10 builds still running > the > >>> CPU usage went down to 5%, stayed there for 5 minutes and all was > smooth and > >>> quick again. From then on it keps oscillating, I've seen spikes of > 240% :( > >>> > >>> > >>> > >>> > >>> > >>> On Tue, Aug 2, 2016 at 4:12 PM, Francesco Di Mizio > >>> <[email protected]> wrote: > >>>> > >>>> Sometimes it goes up to 140%. I was not able to relate this with a > >>>> particular builds condition - seems like it can happen any time and > is not > >>>> related to how many builds are going on. > >>>> > >>>> I usually realize the server got into this state because the web UI > gets > >>>> stuck. As soon as the CPU% goes back to normal values (2-3% most > times) the > >>>> web finishes loading just instantly. > >>>> > >>>> Any pointers as to what might be causing this? Only reason I can think > >>>> of is too many people trying to access the web UI simultaniously - > may I be > >>>> right? > >>>> > >>> > >> > >> _______________________________________________ > >> users mailing list > >> [email protected] > >> https://lists.buildbot.net/mailman/listinfo/users > > > > > > _______________________________________________ > > users mailing list > > [email protected] > > https://lists.buildbot.net/mailman/listinfo/users >
_______________________________________________ users mailing list [email protected] https://lists.buildbot.net/mailman/listinfo/users
