I checked on some of our Jenkins instances in 1.562.2 and didn't get messages about MultiThreadedHttpConnectionManager
How are deployed your Jenkins ? * Java 6/7 ? * Tomcat 7/Jetty as engine ? 2014-09-17 10:31 GMT+02:00 Arnaud Héritier <[email protected]>: > dpkg: warning: downgrading jenkins from 1.565.2 to 1.554.3 #onemoretime > > 1.565.2 unusable again for me. UI too slow or doesn't reply... > I really don't understand the change in jenkins core that may create the > change httpclient integration used by crowd > > > > > On Tue, Sep 16, 2014 at 6:27 PM, Arnaud Héritier <[email protected]> > wrote: >> >> Hi >> >> I just deployed the 1.565.2 >> I hope the slowness I had with 1.565.1 was due to issue 18065 with all >> my maven jobs and the dashboard plugin >> Sadly my logs are always filled by messages like : >> >> Sep 16, 2014 6:24:27 PM >> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager >> freeConnection >> >> SEVERE: Host connection pool not found, >> hostConfig=HostConfiguration[host=https://identity.exoplatform.org] >> >> >> You saw noone else who reported such problem with the crowd plugin ? >> I didn't find anything in Jira but I'm surprised to be the only one to >> have this problem with a Jenkins LTS and the crowd plugin >> >> Cheers >> >> >> On Mon, Aug 4, 2014 at 10:11 AM, Arnaud Héritier <[email protected]> >> wrote: >>> >>> Rolling back to 1.554.3 remove the log. I don't understand which change >>> in core can produce this side-effect on crowd plugin / httpclient >>> >>> >>> On Fri, Aug 1, 2014 at 10:48 AM, Arnaud Héritier <[email protected]> >>> wrote: >>>> >>>> ok sadly this morning I discover that it wasn't a good idea to hide this >>>> problem. My instance is really slow with ~10 threads blocked like this : >>>> >>>> RequestHandlerThread[#700] >>>> >>>> "RequestHandlerThread[#700]" Id=63426 Group=main BLOCKED on >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool@748b8ee1 >>>> owned by "RequestHandlerThread[#701]" Id=63427 >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool.freeConnection(MultiThreadedHttpConnectionManager.java:984) >>>> - blocked on >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool@748b8ee1 >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.releaseConnection(MultiThreadedHttpConnectionManager.java:643) >>>> at >>>> org.apache.commons.httpclient.HttpConnection.releaseConnection(HttpConnection.java:1179) >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.releaseConnection(MultiThreadedHttpConnectionManager.java:1423) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.ensureConnectionRelease(HttpMethodBase.java:2430) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.responseBodyConsumed(HttpMethodBase.java:2422) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase$1.responseConsumed(HttpMethodBase.java:1892) >>>> at >>>> org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java:198) >>>> at >>>> org.apache.commons.httpclient.AutoCloseInputStream.checkClose(AutoCloseInputStream.java:170) >>>> at >>>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:109) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2900) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:302) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1502) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$TrailingMiscDriver.next(XMLDocumentScannerImpl.java:1372) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) >>>> at >>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649) >>>> at >>>> com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:203) >>>> at >>>> com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:181) >>>> at >>>> com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:235) >>>> at javax.xml.bind.JAXB.unmarshal(JAXB.java:223) >>>> at >>>> com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:349) >>>> at >>>> com.atlassian.crowd.integration.rest.service.RestCrowdClient.getGroupsForUser(RestCrowdClient.java:641) >>>> at >>>> de.theit.jenkins.crowd.CrowdConfigurationService.getAuthoritiesForUser(CrowdConfigurationService.java:280) >>>> at >>>> de.theit.jenkins.crowd.CrowdRememberMeServices.autoLogin(CrowdRememberMeServices.java:136) >>>> at >>>> de.theit.jenkins.crowd.CrowdServletFilter.doFilter(CrowdServletFilter.java:181) >>>> at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) >>>> at >>>> org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) >>>> at >>>> hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) >>>> at >>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) >>>> at >>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >>>> at >>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) >>>> at >>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) >>>> at >>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>>> at >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>>> at org.eclipse.jetty.server.Server.handle(Server.java:370) >>>> at >>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) >>>> at >>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) >>>> at >>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) >>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>>> at >>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>>> at >>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>>> at >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) >>>> at >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>>> at >>>> winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> Number of locked synchronizers = 1 >>>> - java.util.concurrent.ThreadPoolExecutor$Worker@4aa15e5e >>>> >>>> >>>> because of >>>> >>>> RequestHandlerThread[#701] >>>> >>>> "RequestHandlerThread[#701]" Id=63427 Group=main RUNNABLE >>>> at java.lang.String.regionMatches(String.java:1340) >>>> at java.lang.String.equalsIgnoreCase(String.java:1090) >>>> at org.apache.commons.httpclient.HttpHost.equals(HttpHost.java:199) >>>> at >>>> org.apache.commons.httpclient.util.LangUtils.equals(LangUtils.java:63) >>>> at >>>> org.apache.commons.httpclient.HostConfiguration.equals(HostConfiguration.java:506) >>>> - locked org.apache.commons.httpclient.HostConfiguration@28793572 >>>> at java.util.HashMap.getEntry(HashMap.java:471) >>>> at java.util.HashMap.get(HashMap.java:421) >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool.getHostPool(MultiThreadedHttpConnectionManager.java:808) >>>> - locked >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool@748b8ee1 >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool.freeConnection(MultiThreadedHttpConnectionManager.java:991) >>>> - locked >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool@748b8ee1 >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.releaseConnection(MultiThreadedHttpConnectionManager.java:643) >>>> at >>>> org.apache.commons.httpclient.HttpConnection.releaseConnection(HttpConnection.java:1179) >>>> at >>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.releaseConnection(MultiThreadedHttpConnectionManager.java:1423) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.ensureConnectionRelease(HttpMethodBase.java:2430) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.responseBodyConsumed(HttpMethodBase.java:2422) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase$1.responseConsumed(HttpMethodBase.java:1892) >>>> at >>>> org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java:198) >>>> at >>>> org.apache.commons.httpclient.AutoCloseInputStream.checkClose(AutoCloseInputStream.java:170) >>>> at >>>> org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:109) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2900) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:302) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1502) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$TrailingMiscDriver.next(XMLDocumentScannerImpl.java:1372) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) >>>> at >>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649) >>>> at >>>> com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:203) >>>> at >>>> com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:181) >>>> at >>>> com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:235) >>>> at javax.xml.bind.JAXB.unmarshal(JAXB.java:223) >>>> at >>>> com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:349) >>>> at >>>> com.atlassian.crowd.integration.rest.service.RestCrowdClient.getGroupsForUser(RestCrowdClient.java:641) >>>> at >>>> de.theit.jenkins.crowd.CrowdConfigurationService.getAuthoritiesForUser(CrowdConfigurationService.java:280) >>>> at >>>> de.theit.jenkins.crowd.CrowdRememberMeServices.autoLogin(CrowdRememberMeServices.java:136) >>>> at >>>> de.theit.jenkins.crowd.CrowdServletFilter.doFilter(CrowdServletFilter.java:181) >>>> at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) >>>> at >>>> org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) >>>> at >>>> hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) >>>> at >>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) >>>> at >>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >>>> at >>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) >>>> at >>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) >>>> at >>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) >>>> at >>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) >>>> at >>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>>> at >>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>>> at org.eclipse.jetty.server.Server.handle(Server.java:370) >>>> at >>>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) >>>> at >>>> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) >>>> at >>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) >>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>>> at >>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>>> at >>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>>> at >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) >>>> at >>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>>> at >>>> winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:744) >>>> >>>> Number of locked synchronizers = 1 >>>> - java.util.concurrent.ThreadPoolExecutor$Worker@5fb81882 >>>> >>>> >>>> >>>> I don't yet see the relation with the jenkins core upgrade (the crowd >>>> plugin didn't change). >>>> >>>> I will see on crowd plugin side if there is something strange. >>>> >>>> cheers >>>> >>>> >>>> >>>> >>>> On Thu, Jul 31, 2014 at 3:22 PM, Arnaud Héritier <[email protected]> >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I don't know if it is due to the upgrade to Jenkins ver. 1.565.1 but >>>>> I'm noticing in my logs a lot of SEVERE errors I didn't see before : >>>>> >>>>> Jul 31, 2014 3:14:32 PM >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager >>>>> freeConnection >>>>> >>>>> SEVERE: Host connection pool not found, >>>>> hostConfig=HostConfiguration[host=https://identity.exoplatform.org] >>>>> >>>>> >>>>> >>>>> It seems they are useless >>>>> (https://issues.apache.org/jira/browse/HTTPCLIENT-799) and due to the >>>>> crowd >>>>> plugin which uses commons http client 3.x >>>>> >>>>> If in jenkins Logs Admin UI I set the logger >>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager to OFF it >>>>> solves my issue but I loose it when I restart jenkins. >>>>> >>>>> I tried to use an init.groovy.d script but it doesn't work (I'm not >>>>> sure why, I see it is executed but logs are here and I'm not a java >>>>> logging >>>>> fan/expert) >>>>> >>>>> Logger multiThreadedHttpConnectionManagerLoggerStopped = >>>>> Logger.getLogger("org.apache.commons.httpclient.MultiThreadedHttpConnectionManager") >>>>> >>>>> multiThreadedHttpConnectionManagerLoggerStopped.setLevel(Level.OFF) >>>>> >>>>> >>>>> LogManager.getLogManager().addLogger(multiThreadedHttpConnectionManagerLoggerStopped) >>>>> >>>>> >>>>> Is there another solution to set the logger level permanently ? >>>>> >>>>> Thx. >>>>> -- >>>>> ----- >>>>> Arnaud Héritier >>>>> http://aheritier.net >>>>> Mail/GTalk: aheritier AT gmail DOT com >>>>> Twitter/Skype : aheritier >>>> >>>> >>>> >>>> >>>> -- >>>> ----- >>>> Arnaud Héritier >>>> http://aheritier.net >>>> Mail/GTalk: aheritier AT gmail DOT com >>>> Twitter/Skype : aheritier >>> >>> >>> >>> >>> -- >>> ----- >>> Arnaud Héritier >>> http://aheritier.net >>> Mail/GTalk: aheritier AT gmail DOT com >>> Twitter/Skype : aheritier >> >> >> >> >> -- >> ----- >> Arnaud Héritier >> http://aheritier.net >> Mail/GTalk: aheritier AT gmail DOT com >> Twitter/Skype : aheritier > > > > > -- > ----- > Arnaud Héritier > http://aheritier.net > Mail/GTalk: aheritier AT gmail DOT com > Twitter/Skype : aheritier > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
