Hello, We're getting complaints about a Jenkins system we manage, where jobs do not always trigger as quickly as developers expect them to. Our observation is that sometimes there can be a several minute wait between an event in Gerrit and the associated job triggering. We can see this scheduling delay even when doing a manual trigger. This seems to sync up with periods of high CPU usage on the Jenkins master; when I've run jstack or visualvm on Jenkins during high CPU periods, I've seen that the majority of it is caused by Gerrit-related threads all stuck in notifyListener(). Sometimes this sorts itself out, other times we've had to restart Jenkins.
Barring misconfiguration of our jobs, a resource issue, or a bug, it looks like the only other thing we could look at is the send/receive worker thread settings in Jenkins. It's not clear (to me) what effect changing these settings should have or why someone might need to change them. Looking at the code, it does look like GerritHandler.notifyListener blocks on whatever reflected method is called for each handled event, but I haven't looked further to see what those methods would be doing. Are these threads consumed for the duration of a triggered build? Should we try to have a receive thread per desired concurrent triggered build? Our organization is supporting several projects with a single Jenkins master and many build agents. A rough estimate based on Jenkins logs over the last 12 months is that we're supporting ~60 triggered jobs per day, which matches up with about that many new patchsets pushed per day to Gerrit based on its logs over the past 6 months. Not all patchsets trigger jobs, but some trigger more than one job. Based on what I've read about with other organizations, this doesn't seem like an especially busy environment. We've currently got the Gerrit trigger receiving threads set to 6 and the sending threads at the default of 1. Does anyone here with a similar environment have some recommendations? Thanks, -Will -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b714b5f1-c846-4495-8ad8-1de891c6bf53%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
