I use LDAP for authentication. I ran ldap in debug mode and could see Jenkins continuously search for all the users in an unexpected manner which causing server to consume almost 100% cpu. Meanwhile, Jenkins logs indicate LDAP response timeouts as attached below.
When I stop the Jenkins server, LDAP does not consume considerable amount of CPU. PID USER PR NI VIRT RES SHR S *%CPU* %MEM TIME+ COMMAND 2825 openldap 20 0 917256 94672 7044 S *99.5* 9.3 0:12.48 slapd *openldap sample debug logs* 582558ca => access_allowed: search access to "uid=testuser,ou=user,dc=ldap,dc=domain,dc=org" "mail" requested 582558ca => dn: [2] ou=group,dc=ldap,dc=domain,dc=org 582558ca => dn: [8] 582558ca => acl_get: [9] attr mail 582558ca => acl_mask: access to entry "uid=testuser,ou=user,dc=ldap,dc=domain,dc=org", attr "mail" requested 582558ca => acl_mask: to value by "uid=ldapbinduser,ou=user,dc=ldap,dc=domain,dc=org", (=0) 582558ca <= check a_dn_pat: uid=ldapbinduser,ou=user,dc=ldap,dc=domain,dc=org 582558ca <= acl_mask: [2] applying read(=rscxd) (stop) 582558ca <= acl_mask: [2] mask: read(=rscxd) 582558ca => slap_access_allowed: search access granted by read(=rscxd) 582558ca => access_allowed: search access granted by read(=rscxd) 582558ca <= test_filter 5 582558ca bdb_search: 648 does not match filter 582558ca => test_filter 582558ca EQUALITY *Jenkins logs* Nov 11, 2016 11:07:04 AM hudson.security.LDAPSecurityRealm$LDAPUserDetailsService loadUserByUsername WARNING: Failed to search LDAP for username=someone org.acegisecurity.ldap.LdapDataAccessException: LdapCallback;LDAP response read timed out, timeout used:60000ms.; nested exception is javax.naming.NamingException: LDAP response read timed out, timeout used:60000ms.; remaining name 'ou=user' at org.acegisecurity.ldap.LdapTemplate$LdapExceptionTranslator.translate(LdapTemplate.java:295) at org.acegisecurity.ldap.LdapTemplate.execute(LdapTemplate.java:128) at org.acegisecurity.ldap.LdapTemplate.searchForSingleEntry(LdapTemplate.java:246) at org.acegisecurity.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:119) at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:708) at hudson.security.LDAPSecurityRealm$LDAPUserDetailsService.loadUserByUsername(LDAPSecurityRealm.java:670) at hudson.security.LDAPSecurityRealm.loadUserByUsername(LDAPSecurityRealm.java:572) at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1050) at hudson.model.User.get(User.java:395) at hudson.model.User.get(User.java:364) at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:288) at hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:349) at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:392) at hudson.model.User.relatedTo(User.java:626) at hudson.model.User.doRssLatest(User.java:820) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:89) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:99) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: javax.naming.NamingException: LDAP response read timed out, timeout used:60000ms.; remaining name 'ou=user' at com.sun.jndi.ldap.Connection.readReply(Connection.java:483) at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639) at com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562) at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1985) at com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1847) at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1772) at com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1789) at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:412) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:394) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:376) at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:286) at org.acegisecurity.ldap.LdapTemplate$3.doInDirContext(LdapTemplate.java:249) at org.acegisecurity.ldap.LdapTemplate.execute(LdapTemplate.java:126) ... 95 more On Thursday, November 10, 2016 at 3:46:54 PM UTC+5:30, Paxton, Darren wrote: > > Suspect you need to provide a lot more information such as what are you > using LDAP for,purely authentication? Do the LDAP logs indicate what the > connections could be? > > > > More info about what you’ve done to troubleshoot yourself. > > > > *From:* [email protected] <javascript:> [mailto: > [email protected] <javascript:>] *On Behalf Of *Rumesh Bandara > *Sent:* 10 November 2016 09:48 > *To:* Jenkins Users > *Subject:* Large number of "ESTABLISHED" LDAP connections initialted by > jenkins > > > > Hi All, > > > > Our Jenkins instance is making a large number of "ESTABLISHED" connections > to ldap server which cause higher cpu usage of ldap instance. Do you have > any clue about what could be the issue from Jenkins? > > > > Jenkins ver. 2.7.2 > > > > Thanks, > > Rumesh > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/d1c81074-c421-4b69-81b8-0d66dbd7c4dc%40googlegroups.com > > <https://groups.google.com/d/msgid/jenkinsci-users/d1c81074-c421-4b69-81b8-0d66dbd7c4dc%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/bb66af19-8d09-4895-bd82-c5b19e916dc8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
