Konstantinos, Could you open a Jira issue for this and attach a patch? This would be helpful for the project.
https://issues.apache.org/jira/browse/OODT/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel Thanks, Paul Ramirez > On Jul 10, 2014, at 5:23 AM, "Konstantinos Mavrommatis" > <[email protected]> wrote: > > Hi, > > Following the suggestion at > > http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/ > > > > I modified the code in > > src/main/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrClient.java > > > > and added the line highlighted in red > (method.setRequestHeader(“Connection”,”close”). > > This seems to have resolved the problem at least until now ingestion of few > thousand files have not produced any stale connection. > > Please let me know your thoughts on this solution. > > Best, > > Konstantinos > > > > ====================================================== > > private String doHttp(HttpMethod method) throws Exception { > > > > StringBuilder response = new StringBuilder(); > > BufferedReader br = null; > > try { > > > > // send request > > HttpClient httpClient = new HttpClient(); > > // 10 July 2014. attempting to avoid problems with > CLOSE_WAIT connections Based on > > > //http://blogs.nuxeo.com/development/2013/02/using-httpclient-properly-avoid-closewait-tcp-connections/ > > method.setRequestHeader("Connection", "close"); > > int statusCode = httpClient.executeMethod(method); > > > > // read response > > if (statusCode != HttpStatus.SC_OK) { > > > > // still consume the response > > method.getResponseBodyAsString(); > > throw new CatalogException("HTTP method failed: " + > method.getStatusLine()); > > > > } else { > > > > // read the response body. > > br = new BufferedReader(new > InputStreamReader(method.getResponseBodyAsStream())); > > String readLine; > > while(((readLine = br.readLine()) != null)) { > > response.append(readLine); > > } > > > > ======================================================= > > > > Information about the system: > > OS is MacOS Mavericks, > > SOLR version : 4.6.1 , SOLR runs on a single server, no replication at all. > > Tomcat verision: 7.0.50 > > OODT version is 0.6 > > Java(TM) SE Runtime Environment (build 1.7.0_51-b13) > > > > > > > >> -----Original Message----- > >> From: Lewis John Mcgibbney [mailto:[email protected]] > >> Sent: Wednesday, July 09, 2014 4:57 AM > >> To: [email protected] > >> Subject: Re: File manager keeps connections to SOLR in CLOSE_WAIT state > >> for hours > > >> It's just occoured to me that everything in > >> org.apache.oodt.cas.filemgr.catalog.solr.SolrCatalog (well, > >> specifically > >> org.apache.oodt.cas.filemgr.catalog.solr.SolrClient) is done through > >> HTTPClient so all of the above may not be relevant. > >> Can you please scope all the same? > >> Thanks > >> Lewis > > > >> On Tue, Jul 8, 2014 at 10:54 PM, Lewis John Mcgibbney < > >> [email protected]<mailto:[email protected]>> wrote: > > >>> Hi Konstantinos, > >>> OK, I was ablel to scope this one and I have a few questions for you. > >>> 1) Which version of Solr are you using? Is it 3.5, 3.6, 4.0-ALPHA? > >> If > >>> so please scope this issue [0], the solution would be to upgrade if > >>> you are not too long ahead with ingestion as fixes in Solr are worth > >>> having based on recent release cycles. > >>> 2) How many cores do you have on Solr server? Also what kind of setUp > >>> do you have? Replication at all? > >>> In recent versions of Solr 4.X SolrJ clients should now call > >>> shutdown() on their SolrServer object to let it know they don't want > >>> to re-use any existing connections anymore, and when Solr internally > >>> uses SolrJ to talk to other nodes in SolrCloud it should be doing > >> this > >>> (as of 4.0-ALPHA)so this is why I ask. > >>> Lewis > > >>> [0] https://issues.apache.org/jira/browse/SOLR-3280 > > > >>> On Tue, Jul 8, 2014 at 7:14 AM, Konstantinos Mavrommatis < > >>> [email protected]<mailto:[email protected]>> wrote: > > >>>> Hi, > >>>> I have setup OODT filemanager on port 9000, using SOLR as the > >>>> indexing service on port 8081. They are both setup on the same > >>>> computer, while crawler runs on a number of different compute nodes > >>>> spread across the local network and the cloud. > > >>>> When the crawler runs and ingests files I notice that there are > >>>> several connections that open to solr and remain in CLOSE_WAIT state > >> for hours. > >>>> any idea why this happens? Moving forward I am planning to use > >>>> several hundreds of crawler instances, each running on different > >>>> computer, that will create thousands of such connections and will > >>>> probably create problems to the system. > >>>> Thanks in advance for any help > >>>> Kostas > > >>>> $lsof -i :8081 > >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF > >> NODE > >>>> NAME > >>>> java 92065 kmavrommatis 75u IPv6 0x392c3fa3b63b29cf 0t0 > >> TCP > >>>> localhost:49205->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 76u IPv6 0x392c3fa3b6dcbbcf 0t0 > >> TCP > >>>> localhost:49206->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 77u IPv6 0x392c3fa39fd12e0f 0t0 > >> TCP > >>>> localhost:49207->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 78u IPv6 0x392c3fa39fdcdbcf 0t0 > >> TCP > >>>> localhost:49208->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 79u IPv6 0x392c3fa3b62cde0f 0t0 > >> TCP > >>>> localhost:49209->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 80u IPv6 0x392c3fa39fa2714f 0t0 > >> TCP > >>>> localhost:49210->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 81u IPv6 0x392c3fa3b6c32acf 0t0 > >> TCP > >>>> localhost:49211->localhost:sunproxyadmin (CLOSE_WAIT) > >>>> java 92065 kmavrommatis 82u IPv6 0x392c3fa3b6aa714f 0t0 > >> TCP > >>>> localhost:49212->localhost:sunproxyadmin (CLOSE_WAIT) > > > >>>> process 92065 is: > >>>> /usr/bin/java -Djava.ext.dirs=../lib > >>>> -Djava.util.logging.config.file=../etc/logging.properties > >>>> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties > >>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager --portNum 9000 > > >>>> ********************************************************* > >>>> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND > >>>> MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE > >>>> OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE. > >>>> If the reader is not the intended recipient, or the employee or > >> agent > >>>> responsible to deliver it to the intended recipient, you are hereby > >>>> notified that any dissemination, distribution or copying of this > >>>> communication is strictly prohibited. If you have received this > >>>> communication in error, please reply to the sender to notify us of > >>>> the error and delete the original message. Thank You. > > > > > >>> -- > >>> *Lewis* > > > > > >> -- > >> *Lewis* > > ********************************************************* > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL > OR INDIVIDUALS NAMED ABOVE. > If the reader is not the intended recipient, or the > employee or agent responsible to deliver it to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this > communication is strictly prohibited. If you have > received this communication in error, please reply to the > sender to notify us of the error and delete the original > message. Thank You.
