If this persists, I would love to get full httpclient log output, as we did before. I am concerned that httpclient 4.2.5, which we adopted in the beginning of last week, has broken our workaround for basic auth. Did you do an ant make-core-deps yesterday, for the first time in a while?
Sent from my Windows Phone -----Original Message----- From: Erlend Garåsen (JIRA) Sent: 4/27/2013 7:12 PM To: [email protected] Subject: [jira] [Commented] (CONNECTORS-679) Web connector seems to be able to get stuck waiting on throttling conditions [ https://issues.apache.org/jira/browse/CONNECTORS-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643831#comment-13643831 ] Erlend Garåsen commented on CONNECTORS-679: ------------------------------------------- The job stopped almost immediate after I started it: {code} ERROR 2013-04-28 00:38:04,539 (Worker thread '5') - Exception tossed: Repeated service interruptions - failure processing document: null org.apache.manifoldcf.core.interfaces.ManifoldCFException: Repeated service interruptions - failure processing document: null at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:586) Caused by: org.apache.http.client.ClientProtocolException at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.manifoldcf.agents.output.solr.ModifiedHttpSolrServer.request(ModifiedHttpSolrServer.java:277) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.manifoldcf.agents.output.solr.HttpPoster$IngestThread.run(HttpPoster.java:885) Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity. at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:693) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ... 6 more {code} Then it got interrupted and stopped (it did not crash). I have restarted the job once more, but now it has been hanging for 25 minutes without any activity in logs etc. The plan now is to deploy the latest trunk on our test server as well in case things perform better. And also start to remove some hosts that may cause problems (such as www.ibsen.uio.no that caused proxy errors). > Web connector seems to be able to get stuck waiting on throttling conditions > ---------------------------------------------------------------------------- > > Key: CONNECTORS-679 > URL: https://issues.apache.org/jira/browse/CONNECTORS-679 > Project: ManifoldCF > Issue Type: Bug > Components: Web connector > Affects Versions: ManifoldCF 1.2 > Reporter: Karl Wright > Assignee: Karl Wright > Priority: Critical > Fix For: ManifoldCF 1.2 > > > While crawling, a production system wound up stuck with the following thread dump: > {code} > 2013-04-24 16:06:52 > Full thread dump Java HotSpot(TM) Server VM (20.13-b02 mixed mode): > "Attach Listener" daemon prio=10 tid=0x09022000 nid=0x2d46 waiting on condition [0x00000000] > java.lang.Thread.State: RUNNABLE > "Thread-78762" daemon prio=10 tid=0xdac6f400 nid=0x5bff in Object.wait() [0xdc7d8000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-76505" daemon prio=10 tid=0x0932e000 nid=0x5304 in Object.wait() [0xdbb18000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-76346" daemon prio=10 tid=0xdc342800 nid=0x5263 in Object.wait() [0xdbbba000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-76181" daemon prio=10 tid=0xde240400 nid=0x51a0 in Object.wait() [0xdb75c000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-76117" daemon prio=10 tid=0xdac01c00 nid=0x5160 in Object.wait() [0xdc4fe000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-76034" daemon prio=10 tid=0xdab05000 nid=0x510d in Object.wait() [0xdc45c000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-75986" daemon prio=10 tid=0xde240c00 nid=0x50dd in Object.wait() [0xdbac7000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-75893" daemon prio=10 tid=0xde2ac000 nid=0x5080 in Object.wait() [0xdbcad000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-75840" daemon prio=10 tid=0xdad54c00 nid=0x504b in Object.wait() [0xdc87a000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-75810" daemon prio=10 tid=0xde23f800 nid=0x502c in Object.wait() [0xdba76000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledConnection.beginRead(ThrottledFetcher.java:1168) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.basicRead(ThrottledFetcher.java:2064) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottledInputstream.read(ThrottledFetcher.java:2032) > at > org.apache.manifoldcf.core.common.XThreadInputStream.stuffQueue(XThreadInputStream.java:86) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ExecuteMethodThread.run(ThrottledFetcher.java:2537) > "Thread-75709" daemon prio=10 tid=0xdde00400 nid=0x4fc7 in Object.wait() [0xdc694000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at > org.apache.manifoldcf.crawler.connectors.webcrawler.ThrottledFetcher$ThrottleBin.beginRead(ThrottledFetcher.java:820) > - locked <0xec11c430> (a java.lang.Integer) > at org.apache.manifoldcf.crawler.connectors.webc
